basetenlabs / truss

The simplest way to serve AI/ML models in production
https://truss.baseten.co
MIT License
917 stars 71 forks source link

Composes trusses to sharing GPUs #242

Closed bolasim closed 1 year ago

bolasim commented 1 year ago

Is your feature request related to a problem? Please describe. After some exploration, it seems like we can definitely composes various trusses into one image so that can share a GPU. This has many advantages for server costs, but also for local development when I want to work with multiple trusses, but my machine only happens to have one GPU.

This is issue is to open up a discussion on what could be the goals for bringing this into truss, both in the near term and long term.

bolasim commented 1 year ago

https://github.com/basetenlabs/truss/tree/main/examples/gpu-sharing