search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
30.71k stars 4.66k forks source link

[Bug]: Assertion `idx < size()' failed (vllm on AMD) #4268

Open Pleias opened 7 months ago

Pleias commented 7 months ago

Your current environment

vllm-0.4.1+rocm573-py3.9-linux-x86_64.egg
compiled from source on an AMD cluster
conda/22.9.0 virtual env.
run with MI250

🐛 Describe the bug

Hi. I've been able to install correctly the latest version of vllm on an AMD cluster.

Yet, just after loading the model I have a low-level bug from llvm:

/root/.triton/llvm/llvm-5e5a22ca-centos-x64/include/llvm/ADT/SmallVector.h:298: const T& llvm::SmallVectorTemplateCommon<T,

::operator[](llvm::SmallVectorTemplateCommon<T, ::size_type) const [with T = long int; = void; llvm::SmallVectorTemplateCommon<T, >::const_reference = const long int&; llvm::SmallVectorTemplateCommon<T, ::size_type = long unsigned int]: Assertion `idx < size()' failed.

I haven't been able to get a detailed traceback. Within vllm the bug comes just after loading the hidden states in model_runner.py:

hidden_states = model_executable(**execute_model_kwargs)

I don't think it's a common issue but could really use at least some pointers.

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!