EmbeddedLLM / vllm-rocm

vLLM: A high-throughput and memory-efficient inference and serving engine for LLMs
https://vllm.readthedocs.io
Apache License 2.0
83 stars 5 forks source link

vLLM >= 0.2.4 with ROCm 5.6.1 #23

Open ekazakos opened 4 months ago

ekazakos commented 4 months ago

Hi,

I'm working in a cluster environment which has ROCm 5.6.1. I have successfully pulled and used your vLLM 0.2.3 docker image; thanks a lot for that! I'm aware that vLLM >= 0.2.4 is meant to work with ROCm 5.7. But I would like to ask you, whether it is possible to use ROCm 5.6.1 for those vLLM versions? Have you tried that? Thank you!