search

zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
7.08k stars 500 forks source link

[Enhancement]: Ability to use the vector-store as a cache #638

Open shashankram opened 1 month ago

shashankram commented 1 month ago

What would you like to be added?

Provide the ability to use the vector store in use for caching as well. For e.g., if I am using Weaviate, I would like to use it for both caching and vector storage.

Thanks

Why is this needed?

No response

Anything else?

No response

SimFG commented 1 month ago

I think this is not a very good idea, because in this caching process, the scalar data will generally be very large, that is, the data size of the questions and their answers is large, and it is not suitable to store them in the vector database, because the advantage of the vector database is vector storage and its search. If many scalars are placed in the vector database, on the one hand, it will not be conducive to retrieval, and on the other hand, it will greatly increase the pressure on the vector database and cause a waste of resources.

shashankram commented 1 month ago

I understand, though the operational complexity with using multiple datastores for caching and vectorization is high. For smaller deployments, I was wondering if using the vector store as a cache for the scalar data would be feasible.

SimFG commented 1 month ago

For this case, you can try using sqlite.