search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.21k stars 3.99k forks source link

[Bug]: Docker image for 0.5.4 does not include package timm==0.9.10 to run MiniCPMV #8107

Open bjornjee opened 2 weeks ago

bjornjee commented 2 weeks ago

Your current environment

nil

🐛 Describe the bug

Running docker image 0.5.4 with the following entrypoint:

python3 -m vllm.entrypoints.openai.api_server --port 8088 --load-format auto --gpu-memory-utilization 0.90 --enforce-eager --disable-log-requests --model HwwwH/MiniCPM-V-2 --served-model-name HwwwH/MiniCPM-V-2 --trust-remote-code --max-model-len 4096 --tensor-parallel-size 1

Server errors out with error, code ref:

install timm==0.9.10

Willing to create MR for fix

Before submitting a new issue...

jeejeelee commented 2 weeks ago

timm needs to be installed locally , rather than as a common module.

bjornjee commented 2 weeks ago

curious if there is a reason why it is not baked in the Dockerfile. For users who run docker image directly, it will be better if the docker image has all required dependencies

jeejeelee commented 2 weeks ago

curious if there is a reason why it is not baked in the Dockerfile. For users who run docker image directly, it will be better if the docker image has all required dependencies

This module is already included in https://github.com/vllm-project/vllm/blob/main/requirements-test.txt. For Dockerfile, if I understand correctly, it should not need to be manually installed again, see: https://github.com/vllm-project/vllm/blob/main/Dockerfile#L124

DarkLight1337 commented 2 weeks ago

curious if there is a reason why it is not baked in the Dockerfile. For users who run docker image directly, it will be better if the docker image has all required dependencies

Different models may have their own dependencies, with some defined on the HuggingFace repo and thus outside of our control. It would be quite inefficient to install the dependencies for every model when you will only use a couple of them.

In any case, it should be easy to create a Dockerfile that adds an extra step on top of the existing one to install the dependency.