Open nchapman opened 7 months ago
Reading about RAG now. So if I get this correctly. You want a table inside the db that contains the document information. Previous to that would would have to add/compile that extension into the sqlite instance so the index can be generated/stored/queried?
Yep that's right! Let's chat about this one live. Some things we should consider:
Since Nick is on vacation this week, this will probably not get done as I need to discuss with him first before building anything @andriikrainii
We need a general vector index for RAG. This should be general purpose and allow different data sets and content to be stored. We don't know the complete schema yet as we aren't quite sure yet how we'll handle external data sources but you can imagine it being something like this:
id
: int or uuid – internal idsource_id
: string? – This would identify a dataset. maybe a group of local files, maybe conversations, or maybe a cloud servicedoc_id
: string – This would be the doc id from the source. This would be used to look up the original data when matcheddata
: string/json – This would be a copy of the data that would be useful during RAG. Large documents likely wouldn't be stored here but their metadata might.embedding
: tbd – This would be the stored vector from the relevant data. There's an open question of whether we should allow multiple vectors per document which might change the database structure. Either we'd have duplicate source/doc ids in a single table or move the embeddings to a separate table. Not sure what the right answer is yet.created_at
updated_at
The workflow would likely be that you would search the index for a matching document and if it was a large document then there would be a separate call to fetch the original data. In the internal use case this would probably look like using WatermelonDB to store the conversations and message which we would then index here. When using the index, we would search for matching documents, get their ids, and then query for the original text via WatermelonDB. We can consider other patterns but this seems like the most flexible way for us to support a lot of different sources that potentially represent a lot of data.
Ideally we'd use sqlite for storing the index. Here's a library to investigate: https://github.com/asg017/sqlite-vss