Prompt Caching by Anthropic Claude

langgenius / dify

Dify is an open-source LLM app development platform. Dify's intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

https://dify.ai

Other

52.45k stars 7.65k forks source link

Prompt Caching by Anthropic Claude #7382

Open amatiytsiv opened 3 months ago

amatiytsiv commented 3 months ago

Self Checks

[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

anthropic announced support for caching it would be nice to have it in model config options: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

2. Additional context or comments

No response

3. Can you help us with this feature?

[ ] I am interested in contributing to this feature.

amatiytsiv commented 2 months ago

any plans on implementing this possibility into dify core and tools as by other tools I can see a significant profit from using these as money saving

dnyg commented 2 months ago

It is a bit of a complicated feature to implement, because cache is not set in model config options - but per prompt.

So we need a way through the UI to specify that a certain prompt should be cached - similar to how you can enable jinja templates with a slider.

I have made a working version that just uses specific tags to trigger caching, but I don't think Dify would be interested in this solution, as it's a poor UX solution

amatiytsiv commented 2 months ago

It is a bit of a complicated feature to implement, because cache is not set in model config options - but per prompt.

So we need a way through the UI to specify that a certain prompt should be cached - similar to how you can enable jinja templates with a slider.

I have made a working version that just uses specific tags to trigger caching, but I don't think Dify would be interested in this solution, as it's a poor UX solution

I believe conversational checkbox next to send button is way to go for now. But most importantly it also has to be implemented as configurable variable in workflow and agents, it can serve as quick job done, avoiding or even preparing good RAG

dosubot[bot] commented 1 week ago

Hi, @amatiytsiv. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

Request to add prompt caching support for Anthropic Claude in model configuration.
Complexity arises as caching is set per prompt, not in model config options.
Discussion on implementing a UI solution, similar to enabling jinja templates.
You suggested a conversational checkbox next to the send button for configuration.

Next Steps

Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!