search

zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
7.06k stars 500 forks source link

[Feature]: support for distributed cache #496

Closed a9raag closed 1 year ago

a9raag commented 1 year ago

Is your feature request related to a problem? Please describe.

As far as I know, in the current implementation, the caching eviction is done in memory using cachetools library. Which makes caching limited to the local system.

Support for the distributed cache using Redis or Memcached will allow gptcache to scale horizontally. the

Describe the solution you'd like.

The initial idea for this would be:

  1. Implement a distributed cache eviction analogous to the current MemoryCacheEviction .
  2. Add support for Redis or Memcached or both to maintain and evict the cache.
  3. Add functionality to load the distributed cache based on the provided configuration

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

a9raag commented 1 year ago

If you guys this seems valid and would work. I can start working on this.

SimFG commented 1 year ago

@a9raag yes, you‘re right! Looking forward to your pr