search

eth-sri / lmql

A language for constraint-guided and efficient LLM programming.
https://lmql.ai
Apache License 2.0
3.48k stars 191 forks source link

Docs for cache behaviour #342

Open benwhalley opened 3 months ago

benwhalley commented 3 months ago

I'm trying to understand how/when LLM calls get cached, especially when using the OpenAI API. I've looked in the docs, but can't find details.

Ideally, in development, I'd like to be able to cache/memoize calls to the API. For example, if one uses a LMQL programe which requests multiple completions, and changes the later part of the programme but leave the early phase unchanged. In this case it seems like the early requests to the API could be cached? This is especially the case if passing a seed which is now supported by the API.

lbeurerkellner commented 2 months ago

This is how caching is actually implemented with OpenAI. However, with sample I think caching does not apply, since it is typically not seeded. With the seed parameter, it could be adapted accordingly.

To enable caching across multiple calls of the same request, make sure to pass a cache="tokens.pkl" parameter.