search

c0sogi / LLMChat

A full-stack Webui implementation of Large Language model, such as ChatGPT or LLaMA.
MIT License
257 stars 45 forks source link

Integrate gptCache #23

Closed Torhamilton closed 1 year ago

Torhamilton commented 1 year ago

Lets save some $$$ by implementing GPTcache There is a docker image and I think it may already work with Redis $ docker pull zilliz/gptcache:latest $ docker run -p 8000:8000 -it zilliz/gptcache:latest

Admin can use temperature settings to bypass cache.

Important: cache must maintain user privacy. Admin can add sitewide cache. This will make FAQ generation a breeze and cost nothing to retrieve cached info each time.

c0sogi commented 1 year ago

Unfortunately, this library does not support redis. Instead, you need to use mysql as a cache, and it only supports 'milvus', 'faiss', 'chromadb', and 'hnswlib' as vector databases.

In my testing with mysql, I found that the library does not support asynchronous operations at all. This means that if multiple users are requesting the cache at the same time, all other users will have to wait for one user to finish until all of them have finished.

Torhamilton commented 1 year ago

Thanks, that was real fast! There is already an open issue for Redis integration https://github.com/zilliztech/GPTCache/issues/271 Let's hope our bet on Redis pays off! I will watch GPTCache