Embedding-based Search

[curran]

As a user, I want to search VizHub by entering keywords or a question.

This search should use vector similarity search

• [x] PoC for pgvector #209
• [ ] Await migration with postgres indexing
• [ ] Use Redis vector similarity search?
• [ ] Use Postgres vector similarity search?
• [ ] Search results page
• [ ] PoC MongoDB similarity search

The results I want to be able to find include:

• [ ] Vizzes
• [ ] Files
• [ ] Users
• [ ] Organizations
• [ ] Folders
• [ ] Comments

[curran]

https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb

[curran]

https://towardsdatascience.com/how-i-turned-my-companys-docs-into-a-searchable-database-with-openai-4f2d34bd8736

[curran]

When it comes to storing vectors in Redis... The ID can contain some information about what kind of thing the vector is attached to

• viz-${info.id}-file-${filename} A file within a Viz - Compute one embedding per file
• user-${user.id} A User - Compute one embedding per user (based on full name, username, bio if available)
• org-${org.id} An Organization - Compute one embedding per org (based on name, username, description, bio if available)
• folder-${folder.id} A Folder - Compute one embedding per folder (based on name, description)

[curran]

UI Inspiration from GitHub for presenting search results of multiple types:

From Observable

[curran]

https://redis.com/blog/how-to-build-a-knowledge-base-platform-using-redis/

[curran]

PoC: MongoDB has vector similarity search!

[curran]

Kinds of results like this:

[curran]

[curran]

[curran]

Search text: folder:@somrOrg/someFolder maps

Filter by:

• Viz (10)
• Users (1)
• Comments (5)
• Datasets (0)

[curran]

• [ ] Code the Search Box functionality
• [ ] Implement Search Results Page for users
• [ ] Integrate Search Results Page for vizzes
• [ ] Enable search within a Profile feature