Closed dannysemi closed 6 months ago
Hey, could you try adding the installation of cupy with the specified version into the worker Dockerfile?
Hey, could you try adding the installation of cupy with the specified version into the worker Dockerfile?
Oh that makes sense. I'll try that now.
I got the same error. I must be doing something wrong. Does this work in your testing?
I haven't tested multi-gpu yet, will be working on fixing this later today
Seems like the quickest way to fix this, if following in vLLM's steps, will be to use the runtime image instead of base like they did in the snippet you shared, but the issue is that it would make the container 1.5GB heavier
Not ideal, but it looks like they didn't have a better solution either.
Also, they added lora support I guess. https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/serving_chat.py#L25
I'm just ignoring it for now with this change: https://github.com/dannysemi/worker-vllm/commit/d90b012a5f7ad78c483ccad7718c4d54c5931d3f#diff-865c0134cc71b80102811a3e1216d5dad097594d0a4cabcb4dfc077d925af689R200
I didn't see a change for that here.
It's weird because it says it is Optional
, but it doesn't run unless I specify None
.
It won't run without that change because of the arguments being positional, so what happens is that chat template is passed as lora_modules argument. The new base images are meant for 0.3.0, where that change is already included.
If you're interested, the openai-sse-output
branch (soon to be released as 0.3.0) has OpenAI compatibility, support for Gemma, general improvements and new features - the docs are up-to-date to get you started. You'll have to enforce eager there too until I fix the multi-gpu issue, but it's available as an env variable.
Awesome! Thanks
If you're interested, the
openai-sse-output
branch (soon to be released as 0.3.0) has OpenAI compatibility, support for Gemma, general improvements and new features - the docs are up-to-date to get you started. You'll have to enforce eager there too until I fix the multi-gpu issue, but it's available as an env variable.
I just tried out the new OpenAI compatibility and it works great. Good job!
Thank you, great to hear that! :)
Let me know if you run into any issues or have feature requests
Fixed Multi-GPU in 0.3.0 release :)
Thanks again. If I was already using this image in a region, will it pull the fixed image without the version being bumped?
Only if the workers are deleted/restarted. We check if there are any updates to an image when initializing workers
I based my custom worker on the vllm-base image
base-0.3.0-cuda12.1.0
but if I try to run it with multiple gpus I get this error:It works fine if I'm only using one gpu. I saw this comment on the sls-worker repo at Dockerfile:
Was this Dockerfile used to create the base image?
I can see cupy listed in the requirements.txt too.