search

basetenlabs / truss-examples

Examples of models deployable with Truss
https://trussml.com
MIT License
130 stars 37 forks source link

Fixed stream=True option for VLLM template #348

Closed derrick-yyang closed 2 weeks ago

derrick-yyang commented 2 weeks ago

Currently when we process model_input, we popped the "stream" property.

We want to maintain this property when openAI compatible mode is enabled so stream works with openAI SDK calls