[Bug]: Qwen2.5-32B-GPTQ-Int4 inference `!!!!!`

[jklj077]

Your current environment

The output of `python collect_env.py`

N/A; happened to multiple users.

Model Input Dumps

No response

🐛 Describe the bug

We have been receiving reports that the 4-bit GPTQ version of Qwen2.5-32B-Instruct cannot be used with vllm. The generation only contains !!!!!. However, it was also reported that the same model worked using transformers and auto_gptq.

Here are some related issues:

• https://github.com/QwenLM/Qwen2.5/issues/945 (v0.6.1.post2, v0.6.2, v0.6.3)
• https://github.com/QwenLM/Qwen2.5/issues/1103 (v0.6.1)
• https://github.com/QwenLM/Qwen2.5/issues/1038 (v0.4.2, v0.5.1)

We attempted to reproduce the issue, which appears related to quantization kernels, and the following is a summary:

• gptq_marlin works
• gptq fails for requests with len(prompt_token_ids)<=50 but works for longer input sequences

The results are consistent for

• tensor-parallel-size: 2, 4, 8
• vllm versions: v0.6.1.post2, v0.6.2, v0.6.3.post1, v0.6.4.post1
• nvidia driver versions: 535.183.06, 560.35.05

As gpt_marlin is not available for turing and volta cards, we are not able to find a workaround for those users. It would help a lot if one could help investigate the issue.

Before submitting a new issue...

• [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

cc @robertgshaw2-neuralmagic

[youqugit]

I encountered the same issue, only the /chat/completions endpoint returns an error that many !!!!!, while the /completions endpoint works fine.

vLLM version: 0.6.1

[DarkLight1337]

Also cc @mgoin

[mgoin]

As far as I can tell the gptq kernel hasn't been touched all year, the last change was https://github.com/vllm-project/vllm/pull/2330 by @chu-tianxiang

This may be a fundamental issue with the kernel for this model, someone would need to dive in and learn about it.