Add vector database and Rust API as a general document index

[nchapman]

We need a general vector index for RAG. This should be general purpose and allow different data sets and content to be stored. We don't know the complete schema yet as we aren't quite sure yet how we'll handle external data sources but you can imagine it being something like this:

• id: int or uuid – internal id
• source_id: string? – This would identify a dataset. maybe a group of local files, maybe conversations, or maybe a cloud service
• doc_id: string – This would be the doc id from the source. This would be used to look up the original data when matched
• data: string/json – This would be a copy of the data that would be useful during RAG. Large documents likely wouldn't be stored here but their metadata might.
• embedding: tbd – This would be the stored vector from the relevant data. There's an open question of whether we should allow multiple vectors per document which might change the database structure. Either we'd have duplicate source/doc ids in a single table or move the embeddings to a separate table. Not sure what the right answer is yet.
• created_at
• updated_at

The workflow would likely be that you would search the index for a matching document and if it was a large document then there would be a separate call to fetch the original data. In the internal use case this would probably look like using WatermelonDB to store the conversations and message which we would then index here. When using the index, we would search for matching documents, get their ids, and then query for the original text via WatermelonDB. We can consider other patterns but this seems like the most flexible way for us to support a lot of different sources that potentially represent a lot of data.

Ideally we'd use sqlite for storing the index. Here's a library to investigate: https://github.com/asg017/sqlite-vss

[ospfranco]

Reading about RAG now. So if I get this correctly. You want a table inside the db that contains the document information. Previous to that would would have to add/compile that extension into the sqlite instance so the index can be generated/stored/queried?

[nchapman]

Yep that's right! Let's chat about this one live. Some things we should consider:

• [ ] Pick an embedding model
• [ ] Support full text indexing
• [ ] Support vector indexing (validate sqlite-vss as a solution)
• [ ] Investigate reranking options to combine full text and vector indexes
• [ ] Outline a few use cases to help validate solution and inform the API

[ospfranco]

Since Nick is on vacation this week, this will probably not get done as I need to discuss with him first before building anything @andriikrainii