zilliztech / GPTCache

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
https://gptcache.readthedocs.io
MIT License
6.89k stars 480 forks source link

[Bug]: cannot reproduce the benchmark result #594

Closed songsey closed 5 months ago

songsey commented 6 months ago

Current Behavior

The benchmark code below does not fail https://github.com/zilliztech/GPTCache/blob/main/examples/benchmark/benchmark_sqlite_faiss_onnx.py

Expected Behavior

if you set the threshold at 1 cache miss rate should be 1.

Steps To Reproduce

1. Run the code as-is. (To make it simple, you may use only the first 20 data points in "mock_data.json"). Set the threshold at 1.0. 
2. Upon checking the result, cache miss ratio is 0. 
3. Delete the cache storage and the index file
4. Repeat. cache miss ratio is  still 0 no matter what threshold is used.

Environment

No response

Anything else?

No response