search

runpod-workers / worker-vllm

The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
MIT License
246 stars 101 forks source link

Are visual LLMs supported? #122

Open iamkhalidbashir opened 1 month ago

iamkhalidbashir commented 1 month ago

I am trying to run https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct on serverless but it throws error

2024-10-09 18:06:06.525
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:50.528
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:34.515
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:05:19.851
[my1jfmdyltrax5]
[error]
worker exited with exit code 1
2024-10-09 18:04:18.168
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:04:01.663
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:45.158
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:28.734
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:03:12.261
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:02:55.743
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
2024-10-09 18:02:39.224
[eau4o046d9uc4d]
[error]
worker exited with exit code 1
cris-almodovar commented 1 month ago

multi-modal models like Qwen2-VL is supported in pods, but it seems not in serverless endpoints. see my comment here: https://github.com/runpod-workers/worker-vllm/issues/114#issuecomment-2412967748

iamkhalidbashir commented 1 month ago

yes, this is fixed in the dev branch of vllm which when released in the next version should be fixed for endpoints as well.

therealadityashankar commented 1 month ago

Hi there!, At the moment we do not support visual-LLMs, but we are working on support for these in the future