search

anthropics / anthropic-cookbook

A collection of notebooks/recipes showcasing some fun and effective ways of using Claude.
MIT License
4.72k stars 505 forks source link

Prompt caching design defect? #89

Closed unicomp21 closed 5 days ago

unicomp21 commented 2 weeks ago

The current approach will waste massive amounts of bandwidth? Those in bandwidth constrained environments (ie starlink) will be at a disadvantage due to the wasted bandwidth? Why not follow the redis evalsha pattern to eliminate redundant payloads and wasted bandwidth?

https://github.com/anthropics/anthropic-sdk-typescript/issues/516

https://redis.io/docs/latest/commands/evalsha/

unicomp21 commented 2 weeks ago

I realize appeal of current approach is lower complexity, perhaps this means we should have two api's? The current approach, and then another secondary api/approach which uses sha hashes to store/lookup the cached prompts? ie If sha lookup fails then http error returns, and prefix/prompt needs to be reloaded?

unicomp21 commented 5 days ago

@alexalbertt what does completed mean? Will we have a bandwidth efficient mechanism like evalsha?