[Feature]: Support Caching with Batch Completions

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]

https://docs.litellm.ai/docs/

Other

14.2k stars 1.68k forks source link

[Feature]: Support Caching with Batch Completions #508

Open krrishdholakia opened 1 year ago

krrishdholakia commented 1 year ago

The Feature

enable caching to work for batch completions

Motivation, pitch

churned user feedback

Twitter / LinkedIn details

No response

psimm commented 8 months ago

Hi, based on https://github.com/BerriAI/litellm/issues/1350 I understand that caching for batch embeddings was implemented. Could the same solution be applied to batch completions?