search

simonw / llm-claude-3

LLM plugin for interacting with the Claude 3 family of models
Apache License 2.0
254 stars 23 forks source link

Prompt caching options #28

Open irthomasthomas opened 2 days ago

irthomasthomas commented 2 days ago

Hi Simon, I'm about to add prompt caching to the new branch and wonder what your thoughts are on the implementation?

Options:

  1. Cache everything automatically.
  2. -o cache to cache both user and system prompt together
  3. -o cache_system and -o cache_prompt to cache either one independently.

I think, given the extra 20% cost to cache inputs, options 2 or 3 makes most sense.

What do you think?

Cheers