search

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
https://bentoml.com
Apache License 2.0
9.73k stars 620 forks source link

Fixes Deprecation warning for PyTorch #938

Closed shubh1777 closed 3 months ago

shubh1777 commented 5 months ago

PR Title: Update Docker Run Command to Use vLLM Backend

Description:

This PR addresses issue #936 , which involves deprecation warnings when using the PyTorch backend with the ghcr.io/bentoml/openllm Docker image. The solution updates the Docker run command to utilize the vLLM backend, resolving the deprecation issue and ensuring compatibility with future releases.

Updated Command:


docker run --rm -it -p 3000:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend vllm