Closed elabz closed 2 months ago
One more detail - Docker container was started as
docker run -d \
--shm-size=10.24gb \
--gpus '"device=2"' \
-v /data/models:/root/.cache/huggingface \
--env "HF_TOKEN=ma_token" \
-p 8000:8000 \
--restart unless-stopped \
--name vllm-openai \
vllm/vllm-openai \
--host 0.0.0.0 \
--port 8000 \
--model=astronomer/Llama-3-8B-Instruct-GPTQ-4-Bit \
--enforce-eager \
--dtype=half \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size=1
I am closing it myself - the issue appears to be in loading the model - the python script never gets to the uvicorn
section, so the actual server on 8000 never starts, even those the python script is running
Hello ! I have the same issue... Did you solve it ? I am very interested if you have any solution :) Thank you !
Hello. I am facing a similar issue. Is there a solution for this? Thanks
Your current environment
Please note this is inside a Docker container built locally, so the collect_env.py was run as
docker exec -it vllm-openai sh -c '/usr/bin/python3 /vllm-workspace/collect_env.py'
Verify Python process is working inside the container:
install nettools:
docker exec -it vllm-openai apt-get install net-tools
verify ports open on container:
It listens, just not on 8000 despite the
--port 8000
directive in the commandDocker has port 8000 open and forwarded:
According to the container logs the model appears to be loading (which is also confirmed by running nvtop and seeing VRAM memory usage and GPU activity
docker logs -f vllm-openai
🐛 Describe the bug
try getting any output from the API, get Connection Reset by Peer error:
Any help or nudge in the right direction will be greatly appreciated!