search

basetenlabs / truss-examples

Examples of models deployable with Truss
https://trussml.com
MIT License
103 stars 24 forks source link

added gemma2 9b and 27b with streaming using local-gemma #318

Open dsingal0 opened 4 days ago

dsingal0 commented 4 days ago

vllm's next release will add support for gemma2 9/27B. Until then you'd have to build from source on top of a pytorch image which takes 30+ minutes to deploy. https://github.com/vllm-project/vllm/issues/5806