Open iostreams opened 10 months ago
@iostreams - not that easy. Bedrock is made more for command line tools and python SDKs. I wish it was as easy to use as most of the other APIs.
Hi @enricoros @iostreams - I believe we can make this easier I’m the maintainer of LiteLLM - we allow you to deploy an LLM proxy to call 100+ LLMs in 1 format - Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.
If this looks useful (we're used in production)- please let me know how we can help.
Bedrock request
curl http://0.0.0.0:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "bedrock/anthropic.claude-instant-v1",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
gpt-3.5-turbo request
curl http://0.0.0.0:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
claude-2 request
curl http://0.0.0.0:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "claude-2",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
Thanks @ishaan-jaff, love it. If I understand correctly, we can use LiteLLM in proxy mode, and treat it like OpenAI.
As such I could add a LiteLLM-Proxy vendor in Big-AGI:
Is this correct?
if you're using the deployed proxy here's how you can make a request, with the deployed endpoint
curl https://litellm-7yjrj3ha2q-uc.a.run.app/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
yes - you understood it correctly. I followed up for Linkedin with more details
Added to the roadmap, and to @fredliubojin visibility. I believe he has made some progress already.
Any progress here @enricoros ? Just the LiteLLM connection would be very helpfull for use with Amazon Bedrock (Claude V2, Cohere etc.).
Please add support for Amazon Bedrock. This will allow direct access to Anthropic's models. Amazon Bedrock provides access to the LLMs Amazon Titan, Stable Diffusion XL, Meta Llama 2 (coming soon), AI21 Labs Jurassic, Cohere Command.
Amazon Bedrock Documentation