search

basetenlabs / truss-examples

Examples of models deployable with Truss
https://trussml.com
MIT License
130 stars 37 forks source link

update vllm to 0.6.0 for llama 3.1 #344

Closed zhyncs closed 2 weeks ago

zhyncs commented 3 weeks ago

As titled. I've tested llama/llama-3_1-8b-instruct on https://app.baseten.co/. Since I have only tested Llama 3.1, I have only made changes to these. I may upgrade the vllm versions of other models in the future.

Hi @pankajroark @philipkiely-baseten May you help review this PR? Thanks.

zhyncs commented 2 weeks ago

I will submit another PR to update the vLLM version for all models later on.