search

danny-avila / LibreChat

Enhanced ChatGPT Clone: Features OpenAI, Assistants API, Azure, Groq, GPT-4 Vision, Mistral, Bing, Anthropic, OpenRouter, Vertex AI, Gemini, AI model switching, message search, langchain, DALL-E-3, ChatGPT Plugins, OpenAI Functions, Secure Multi-User System, Presets, completely open-source for self-hosting. More features in development
https://librechat.ai/
MIT License
15.28k stars 2.54k forks source link

Enhancement: Amazon Bedrock Support #2620

Open nacartwright opened 2 months ago

nacartwright commented 2 months ago

What features would you like to see added?

I'd like to see support for Amazon Bedrock. I have several customers that are looking to leverage models in Bedrock to give their users a ChatGPT like experience. LibreChat would be an awesome option for this.

More details

Support for Amazon Bedrock and KB would be an awesome feature.

Which components are impacted by your request?

No response

Pictures

No response

Code of Conduct

danny-avila commented 2 months ago

Yup, adding native bedrock support is planned!

bsu3338 commented 2 months ago

You may could use LiteLLM as a temporary workaround.

https://docs.litellm.ai/docs/providers/bedrock

khfung commented 1 week ago

anthropic keeps banning account for no reason, hopefully bedrock support is coming soon

marty-sullivan commented 1 week ago

Hello, we are planning on using LibreChat at a wide scale at Cornell University. We have a requirement to use Bedrock and have a workaround at the moment, however we want to understand the current state of this native integration. We may be able to contribute if it's just a matter of priorities being elsewhere. However, if there are technical roadblocks preventing this enhancement from being implemented at the moment, it would be good to know that so that we can focus on the workaround for now.

Thanks, Marty Sullivan

danny-avila commented 1 week ago

Hi @marty-sullivan that's exciting!

We may be able to contribute if it's just a matter of priorities being elsewhere.

It is just a matter of priorities being elsewhere. I'm hoping to get to it early July. My idea is that it should be a separate endpoint, similar to Azure OpenAI vs. OpenAI

Also, correct me if I'm wrong, I think Anthropic models have a different API than the other bedrock models (and there may be more differences if Mistral or other big providers are in the same boat).

For this reason, I was thinking of making "Anthropic - Bedrock", "Mistral - Bedrock" their own endpoints, resembling Azure OpenAI vs. OpenAI as mentioned before.

marty-sullivan commented 1 week ago

@danny-avila Thanks for the response! I believe that the API call itself in Bedrock is the same for all the models, however, there is obviously a significant difference in the configuration that needs to be passed to the BedrockRuntime.invoke_model / BedrockRuntime.invoke_model_with_response_stream via the body parameter.

Given that, your idea probably makes sense to maybe have an endpoint for each model, rather than trying to abstract that, but not positive.

Our priorities are in other places right now as well, but your timeline may align well with ours. Perhaps we could sync up on where things are on the week of 7/8 and if there is anywhere we can help we can assess then.