🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.
Ability to customise cache expiration age.
Suggested api:
On specific prompt - cache: {ttl: {days: 30}}
Allowed values: {ttl: {hours, days, minutes}}
Sdk sums up and converts them to ms.
To keep it backwards compatible I’m thinking if cache: true it defaults to {ttl: {days: 3}} when true is specified.
Not sure if there is invalidation api already if one doesn’t exists it might be required.
Use-Case
We have certain queries that are used to generate content in our system. There is no need to run them again every time user requests the same content. Moreover, unless model has been changed or the prompt itself there should be no need to run them again at all.
As mentioned in our discussion current cache implementation resets age every time content is accessed which is perfect for our use case. We don’t necessarily want to store years of content only accessed once
Is this a feature you are interested in implementing yourself?
Proposal
Ability to customise cache expiration age. Suggested api: On specific prompt -
cache: {ttl: {days: 30}}Allowed values:{ttl: {hours, days, minutes}}Sdk sums up and converts them to ms.To keep it backwards compatible I’m thinking if
cache: trueit defaults to{ttl: {days: 3}}when true is specified.Not sure if there is invalidation api already if one doesn’t exists it might be required.
Use-Case
We have certain queries that are used to generate content in our system. There is no need to run them again every time user requests the same content. Moreover, unless model has been changed or the prompt itself there should be no need to run them again at all. As mentioned in our discussion current cache implementation resets age every time content is accessed which is perfect for our use case. We don’t necessarily want to store years of content only accessed once
Is this a feature you are interested in implementing yourself?
Maybe