search

danny-avila / LibreChat

Enhanced ChatGPT Clone: Features Anthropic, AWS, OpenAI, Assistants API, Azure, Groq, o1, GPT-4o, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, langchain, DALL-E-3, ChatGPT Plugins, OpenAI Functions, Secure Multi-User System, Presets, completely open-source for self-hosting. Actively in public development.
https://librechat.ai/
MIT License
19.26k stars 3.21k forks source link

Enhancement: openrouter.ai has prompt caching now #3953

Open FeepingCreature opened 2 months ago

FeepingCreature commented 2 months ago

What features would you like to see added?

https://openrouter.ai/docs/prompt-caching

Implemented for Claude 3 only, but it's there! It would be great to have this available in LibreChat. I frequently run very long conversations that end up very expensive per query; I'd love to be able to cache.

More details

-

Which components are impacted by your request?

No response

Pictures

No response

Code of Conduct

danny-avila commented 2 months ago

Sure I will look into it, but for now, are you not able to use the Anthropic API natively since prompt caching is already implemented for Anthropic?

https://docs.anthropic.com/en/api/getting-started

https://www.librechat.ai/docs/configuration/pre_configured_ai/anthropic

FeepingCreature commented 2 months ago

Well, originally they wanted my phone number and I was like "f*ck that"

Now they want my organization. I am not a corporation, I am a free man! :P

Openrouter is the only api I've found yet that just lets me transact money for API access 1:1. None of the sites seem to care about private users accessing the API privately, so.... meh. They can have my business whenever they make it as easy as "money go in, api come out".

danny-avila commented 2 months ago

Openrouter is the only api I've found yet that just lets me transact money for API access 1:1. None of the sites seem to care about private users accessing the API privately, so.... meh. They can have my business whenever they make it as easy as "money go in, api come out"

I would personally trust Anthropic over Openrouter as a middleman, the mobile number is just an anti-spam prevention method.

In any case, I wanted to highlight it since you can start saving money already, but I will look into it for openrouter.

FeepingCreature commented 2 months ago

I mean, I'm doing code with it. I'm not putting in anything I don't expect people to read. If I cared about privacy more than silly hoops in my usecases, I'd fully agree.

Thank you! I can work without it but it's a bit expensive in long sessions.

lazydogP commented 2 months ago

Is there any progress? I use OpenRouter just because Claude banned my working accounts for no reason, and several new accounts the moment I registered...

wwicak commented 1 month ago

paging @danny-avila for this feature, please add prompt caching for openrouter

prasanthkn83 commented 3 weeks ago

@danny-avila It will be great if this feature can be added

njbbaer commented 1 week ago

@danny-avila Adding my voice, prompt caching for OpenRouter would be very appreciated by me!

avimar commented 22 hours ago

Hi - also would appreciate prompt caching via openrouter.

I'm using everything in a business context, so one payment/receipt at openrouter is much simpler than paying google, openai, anthropic, mistral, etc... to test everything. Just ran out of credit at Anthropic...