search

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
20.79k stars 3k forks source link

Implement semantic cache #5924

Open Rtoribiog opened 3 months ago

Rtoribiog commented 3 months ago

That's a feature that already exists in langchain and will be beneficial to save costs. The idea will be to ported from phyton to c#

https://github.com/zilliztech/GPTCache

https://python.langchain.com/docs/integrations/llms/llm_caching/

https://www.mongodb.com/developer/products/atlas/advanced-rag-langchain-mongodb/

Be able to save all the queries together with llm responses in some database and try to fetch it first from there, if not then call the LLM, and have some parameters to invalidate or update cache from time to time.

matthewbolanos commented 2 months ago

@markwallace-microsoft, going to assign this to @dmytrostruk since this is one of the samples he's working on using filters.

matthewbolanos commented 2 months ago

Also assigning @eavanvalkenburg since he was working on filters for Python

dmytrostruk commented 6 days ago

Example for Semantic Caching in .NET is here: https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Caching/SemanticCachingWithFilters.cs

Rtoribiog commented 5 days ago

Nice, we are going to take a look. Thanks

moonbox3 commented 5 days ago

Removing the .Net tag, since the sample exists. Leaving python as we haven't completed this yet.