runpod-workers / worker-vllm

The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
MIT License
220 stars 85 forks source link

Support custom vLLM build #10

Closed viktor-ferenczi closed 8 months ago

viktor-ferenczi commented 11 months ago

Since vLLM is developing real fast it would be useful to have a command line option (--build-arg) to point to a locally built vLLM package to use instead of a specific version. It would allow for using local builds from main or PR branches, also for using modified versions of vLLM.

nicolasembleton commented 9 months ago

You can just update builder/requirements.txt to suit your needs, or update the builder/setup.sh to check if there's a vLLM package to build and build it with ENV args if there is. It wouldn't be hard to pull of. (but I am not sure I'd see a particular use case personally)

alpayariyak commented 8 months ago

The worker currently uses our modified fork of vllm, to make sure it works with serverless. I'm not sure if adding something like this will create more confusion than utility for the general user base