Support for Amazon Bedrock Routers

lm-sys / RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!

Apache License 2.0

2.78k stars 204 forks source link

Support for Amazon Bedrock Routers #3

Closed dferguson992 closed 1 month ago

dferguson992 commented 1 month ago

Add router support for models hosted on Amazon Bedrock.

iojw commented 1 month ago

Hi there! We support any provider that has a OpenAI-compatible interface, including Amazon Bedrock. You can view instructions for setting up an OpenAI-compatible interface for Amazon Bedrock models here: https://github.com/aws-samples/bedrock-access-gateway

Let me know if you have any questions setting it up.

iojw commented 1 month ago

@dferguson992 We are looking to add LiteLLM support very soon (https://github.com/lm-sys/RouteLLM/pull/11), which should make it even easier to integrate Amazon Bedrock. Instead of having to deploy a CF stack, you now only need to set the API keys for your account. Let us know if you have any thoughts.