Closed behrica closed 1 year ago
Was thinking about it. Cost and performance being main motivations.
For inspiration https://github.com/zilliztech/GPTCache
If you merge #17 , the user can pass a completion function. So this function could as well implement caching. Is this good enough ?
In any case we needed to make the caching plugable, if implemented her.
As they often cost money or are slow, we might want to have a configurable caching (a user supplied cache function ? maybe a default impl)
Should get easier when custom model functions are implemented #8