Open m-revetria opened 1 month ago
@m-revetria caching for llms and chat models is not supported on the stream path at the moment
Hi @eyurtsev, I thought cache was working for the streaming API because the first call to astream
returns the answer from LLM in multiple chunks (token by token), while the second call return the same answer in a single chunk. Does this mean the answer was cached?
is caching for streaming in the roadmap? If so, could you tell us a possible ETA for this feature, please?
Thanks!
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
This is the output generated running the shared code:
Notice the two lines starting with
[MyLogger::async_log_success_event]
sayingcache_hit: 'None'
. It's expected to beTrue
in the second line as the the call toastream
generated a single chunk with the entire message.Description
I'm trying to cache LLM responses using LiteLLM router cache settings and get notified when a response was obtained from cache instead of LLM. For that purpose I've implemented a custom logger as shown in LiteLLM docs.
The issue is that when I call
astream
API, as shown in the code snippet above, thecache_hit
flag isNone
, even though for the case where response is returned from cache.When I call the
ainvoke
API (await llm.ainvoke(...)
) thecache_hit
flag is passed asTrue
to my custom logger as expected after the second call toainvoke
.System Info