vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
26.63k stars 3.9k forks source link

[Feature]: Slurm run_cluster.sh launcher instead of just Ray #7933

Open OrenLeung opened 2 weeks ago

OrenLeung commented 2 weeks ago

🚀 The feature, motivation and pitch

As much as I like Ray, is there any plans to support launching with SLURM too? on most clusters, only have slurm & aren't ray clusters

Alternatives

continue only supporting ray with this script

Additional context

No response

Before submitting a new issue...

simon-mo commented 2 weeks ago

It is possible to run Ray on Slurm using this guide: https://docs.ray.io/en/latest/cluster/vms/user-guides/community/slurm.html