[Feature Request] Support Amazon Bedrock (including Claude and Stable Diffusion XL)

[iostreams]

Please add support for Amazon Bedrock. This will allow direct access to Anthropic's models. Amazon Bedrock provides access to the LLMs Amazon Titan, Stable Diffusion XL, Meta Llama 2 (coming soon), AI21 Labs Jurassic, Cohere Command.

Amazon Bedrock Documentation

[enricoros]

@iostreams - not that easy. Bedrock is made more for command line tools and python SDKs. I wish it was as easy to use as most of the other APIs.

[ishaan-jaff]

Hi @enricoros @iostreams - I believe we can make this easier I’m the maintainer of LiteLLM - we allow you to deploy an LLM proxy to call 100+ LLMs in 1 format - Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.

If this looks useful (we're used in production)- please let me know how we can help.

Usage

Bedrock request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "bedrock/anthropic.claude-instant-v1",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

gpt-3.5-turbo request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

claude-2 request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "claude-2",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

[enricoros]

Thanks @ishaan-jaff, love it. If I understand correctly, we can use LiteLLM in proxy mode, and treat it like OpenAI.

• localhost functionality: ✅
• remote: one needs to specify the IP address of a LiteLLM proxy server ❓ context: big-AGI is deployed serverless and written in typescript, so the proxy operation is the best, and the proxy needs to be accessed via REST from the server (e.g. localhost, or some vercel server)

As such I could add a LiteLLM-Proxy vendor in Big-AGI:

• UI settings:
  • switch (local / remote)
  • remote host (if remote is selected)
• functionality:
  • models will be dynamically discovered by calling the /v1/models endpoint in the LiteLLM server
  • straight 'openai' protocol can be used for streaming and immediate generation

Is this correct?

[ishaan-jaff]

if you're using the deployed proxy here's how you can make a request, with the deployed endpoint

curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

yes - you understood it correctly. I followed up for Linkedin with more details

[enricoros]

Added to the roadmap, and to @fredliubojin visibility. I believe he has made some progress already.

[flirian26]

Any progress here @enricoros ? Just the LiteLLM connection would be very helpfull for use with Amazon Bedrock (Claude V2, Cohere etc.).