Universal semantic search

[dmurfet]

At the end of 2021 we started switching to metaboards being persistent by default. We now have hundreds of boards of mathematics spread across dozens of pockets. That will expand as we increase the number of seminars and attendees. The lesson from the Internet and Google is that one of the few scalable ways of organising information is search. Currently there is no search within metauni. Things we might want to find:

• Particular boards that you remember from a seminar
• A particular calculation you were doing with someone, from which you remember a snippet or a term
• A pocket that you tagged with some labels
• Content that is within a replay

Most of the content you might want to search is on boards. This can be made available to text search via OCR, but the error rate is high enough (and handwriting and positioning random enough) that exact keyword search is unlikely to be very useful. However, we can do better with "semantic search" based on e.g. the OpenAI embeddings (https://beta.openai.com/docs/guides/embeddings). These embeddings transform board OCR text (and transcripts from replays and seminars) into vectors, using the same tech that underlies GPT3 (the vectors are the "thought vector" that is in GPT3's mind when it thinks about that term, poetically speaking).

The vision would be that everything at metauni is accessible to this kind of search, i.e. universal semantic search.

To enable this we need to do a few things

• Generate, store and update board OCR
• Generate and store transcripts of replay transcript, and keep information syncing that text to timing positions in the replay
• Generate and store transcripts of seminars and classes

This data can be then be run through an automated process that generates embedding vectors (running on the GCP VM, it may take time). Semantic search works by taking a query string, generating a vector from it, and then taking dot products with embedding vectors generated from the above process in order to find hits.

[dmurfet]

Features that are enabled by this:

• Web-browser based search of metauni boards, which gives you results and links that let you jump to a particular board
• Search through replays, find a particular search term ("Proposition 30 from Euclid's Elements") or concept, jump to that exact point in the reply and get a URL that you can share to bring other people with you

[dmurfet]

• AI agents within metauni (e.g. the ones commenting on boards) can also pro-actively study a board, look at it's embeddings, and find related boards or replays that it can suggest. "It looks like you're talking about density of states, here's an in-world link that will take you to the seminar where that was discussed"