search

zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
6.89k stars 480 forks source link

[Bug]: benchmark script #600

Open songsey opened 5 months ago

songsey commented 5 months ago

Current Behavior

The llm adapter does not fail but it only to generate directly from llm when the similarity threshold is close enough to the maximum distance. It means what you get is either a positive or a negative hit. There is no cache miss. So I'm not sure what it is trying to do with try-except block in the code below ?

https://github.com/zilliztech/GPTCache/blob/main/examples/benchmark/benchmark_sqlite_faiss_onnx.py

Can you update for the benchmark code that you used to generate the figures in the report in the examples folder ?

Expected Behavior

No response

Steps To Reproduce

No response

Environment

No response

Anything else?

No response

SimFG commented 5 months ago

@songsey You can try to reduce the similarity_threshold, such as 0.9 or 0.8, you will get different results.

songsey commented 5 months ago

It doesn’t seem to me llm adapter is going to raise an exception if it doesn’t meet the theshold requirement?

SimFG commented 5 months ago

Got it. When running this script, the network should be disconnected, which means that when you need to access openai, an error will be reported, which means that the cache is invalid.

songsey commented 5 months ago

Can you check the script with LangChainLLM class ? It doesn’t seem to “disconnect”