Open LGLG42 opened 2 months ago
Encountered the same issue for CPU when running VLLM_CPU_KVCACHE_SPACE=48 python offline_inference.py
with the model Qwen2-72B
:
[rank0]: TypeError: FlashAttentionMetadata.init() got an unexpected keyword argument 'is_prompt'
I'm not positive but I believe this was introduced in https://github.com/vllm-project/vllm/commit/65bf2ac165734fb6339210c4b2b8ce68d2391b77#diff-f81ae2354d4f46c8591a009289e97ab465771d5593feb114539f7cdc58486663L159
Looks like some refactoring occurred here, removing is_prompt
but missing an occurrence in cpu_model_runner.py
. I found this issue seeing CPU inferencing failing in recent versions of vllm. Reverting to 0.4.2 (before this change) may fix the issue.
cc @rkooo567 as the author of that change
I've encountered same issue. when I run following command in vllm gpu backend.
VLLM_CPU_KVCACHE_SPACE=16 python benchmarks/benchmark_throughput.py --model mosaicml/mpt-7b --input-len 128 --output-len 512 --trust-remote-code --backend=vllm --device cpu --dtype bfloat16
--device cpu
my vllm environments was built from source with editable package mode from the main branch.(pip install -e .
)
Here's what I've found.
vllm
Package and Device Type in Backend Selectionvllm
package has separate Python packages for GPU and CPU versions.LLMEngine -> _get_executor_cls -> CPUExecutor -> CPUWorker -> CPU Model Runner -> get_attn_backend
), the get_attn_backend
function determines the backend based on whether the vllm
package is compiled for CPU or GPU, rather than checking the runtime device_type
.vllm
package is GPU-based, even when the CPUExecutor is being used.vllm
package is GPU-based, the CPUExecutor
should not be invoked at all. But I'm not sure when to raise an error if we should do._get_executor_cls
raise an exception when utils.is_cpu
, utils.is_openvino
, etc., do not match the expected device type?WDYT @WoosukKwon ?
Facing the same issue on aarch64 GH200. @anencore94 did you find a solution? I just want to run benchmark_throughput with cpu
Your current environment
π Describe the bug
Running
benchmark_throughput.py ... --device cpu
throws an exception, it works with GPU.Originally I got this error with flash attention as attn backend, but then I played with different versions of vllm and flash, and eventually uninstalled it, and same issue. Error was:
TypeError: FlashAttentionMetadata.__init__() got an unexpected keyword argument 'is_prompt'
I've also tried with other vllm-enabled models, same issue. I've pulled latest main to get updated version of /benchmark_throughput.py, same issue.
My main guess is atm that cpu_model_runner.py and model_runner.py have diverged when calling "attn_metadata = self.attn_backend.make_metadata(" , somehow, somewhere the "is_prompt" kwarg was removed for GPU but not for CPU. I've looked a bit at the code but does not seem to be a trivial fix, so I'll let someone with more experience/time to look into this.