search

deepset-ai / haystack-core-integrations

Additional packages (components, document stores and the likes) to extend the capabilities of Haystack version 2.0 and onwards
https://haystack.deepset.ai
Apache License 2.0
100 stars 96 forks source link

Support prompt caching in Anthropic generators #1004

Open julian-risch opened 3 weeks ago

julian-risch commented 3 weeks ago

Is your feature request related to a problem? Please describe. Anthropic, in particular the models Claude 3.5 Sonnet and Claude 3 Haiku support prompt caching or context caching. Support for Claude 3 Opus is coming soon. We should enable users to use that feature through Haystack to reduce costs and latency. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching https://www.anthropic.com/news/prompt-caching

Describe the solution you'd like We need to implement support for Anthropic's cache_control parameter in messages.

julian-risch commented 3 weeks ago

related issue https://github.com/deepset-ai/haystack-core-integrations/issues/1003

vblagoje commented 1 week ago

To have a full benefit of prompt caching in Haystack pipelines we need to resolve https://github.com/deepset-ai/haystack-core-integrations/issues/1011