feat(API): add light support for batch inference

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

https://bentoml.com

Apache License 2.0

9.39k stars 597 forks source link

Closed aarnphm closed 1 month ago

aarnphm commented 1 month ago