zmedelis / bosquet

Tooling to build LLM applications: prompt templating and composition, agents, LLM memory, and other instruments for builders of AI applications.
https://zmedelis.github.io/bosquet/
Eclipse Public License 1.0
280 stars 19 forks source link

new feature : allow to cache calls to llms #11

Closed behrica closed 1 year ago

behrica commented 1 year ago

As they often cost money or are slow, we might want to have a configurable caching (a user supplied cache function ? maybe a default impl)

Should get easier when custom model functions are implemented #8

zmedelis commented 1 year ago

Was thinking about it. Cost and performance being main motivations.

For inspiration https://github.com/zilliztech/GPTCache

behrica commented 1 year ago

If you merge #17 , the user can pass a completion function. So this function could as well implement caching. Is this good enough ?

In any case we needed to make the caching plugable, if implemented her.