dstack is an open-source alternative to Kubernetes, designed to simplify development, training, and deployment of AI across any cloud or on-prem. It supports NVIDIA, AMD, and TPU.
Some users need to work with both custom and off-the-shelf language models. Deploying off-the-shelf models with dstack may be less convenient and financially viable than using platforms such as AWS Bedrock or Vertex AI, which provide models as a service. In addition, some proprietary models are only available through MaaS platforms. This results in users having to switch back and forth between dstack and MaaS platforms.
Solution
Add support for MaaS platforms starting with AWS Bedrock. Add new router configurations that will allow Bedrock models to be exposed through the dstack-gateway OpenAI-compatible API.
Workaround
Users can use MaaS platforms directly or use proxy solutions to bring MaaS models and models deployed by dstack into a single interface.
Implementation Steps
[ ] Storing data about models in dstack-gateway
[ ] Internal API for model management in dstack-gateway
[ ] Bedrock implementation in dstack-gateway
[ ] MaaS-specific settings in backend configuration
[ ] Storing data about models and routers in dstack-server
[ ] dstack-server routers API
[ ] Routers Python API
[ ] dstack apply router configurations
[ ] dstack router list and dstack router delete commands
Problem
Some users need to work with both custom and off-the-shelf language models. Deploying off-the-shelf models with
dstack
may be less convenient and financially viable than using platforms such as AWS Bedrock or Vertex AI, which provide models as a service. In addition, some proprietary models are only available through MaaS platforms. This results in users having to switch back and forth betweendstack
and MaaS platforms.Solution
Add support for MaaS platforms starting with AWS Bedrock. Add new
router
configurations that will allow Bedrock models to be exposed through thedstack-gateway
OpenAI-compatible API.Workaround
Users can use MaaS platforms directly or use proxy solutions to bring MaaS models and models deployed by
dstack
into a single interface.Implementation Steps
dstack-gateway
dstack-gateway
dstack-gateway
dstack-server
dstack-server
routers APIdstack apply
router configurationsdstack router list
anddstack router delete
commands