TanGentleman / Augmenta

Automate RAG-powered workflows
MIT License
1 stars 0 forks source link

Loading vectorstore doesn't update rag_settings #12

Closed TanGentleman closed 2 months ago

TanGentleman commented 2 months ago

If I have a vectorstore that was indexed in a prior run, then loading it by reading the collection_name in manifest.json works like a charm.

However, this overrides chunk_size and anything involving in document splitting, since the docs have been indexed in the past. To resolve the mismatch, if a vectorstore is loaded, it should update rag_settings with the important fields (chunk_size, chunk_overlap, embedding model). There should be some clear print statements or warnings when there's an incongruency.

TanGentleman commented 2 months ago

Should be fixing a lot of these incongruencies in the coming patch(es)

TanGentleman commented 2 months ago

I decided that this should be modified even before the vectorstore is loaded. This should now be performed whenever the Config class is instantiated.