[BUG] <title>量化后推理报错

[Stark-zheng]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• [X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

微调量化后推理报错

推理代码： from transformers import AutoTokenizer, TextGenerationPipeline, AutoModelForCausalLM

quantized_model_dir = "xxx_int4"

model = AutoModelForCausalLM.from_pretrained( quantized_model_dir, device_map="auto", trust_remote_code=True ).eval()

Either a local path or an url between tags.

image_path = 'xxx/image1.png' response, history = model.chat(tokenizer, query=f'{image_path}这是什么', history=None) print(response)

错误： OSError: /datadisk2/_1009_qlora_ds_adddataset_cleanv1_int4 does not appear to have a file named modeling_qwen.py. Checkout 'https://huggingface.co//datadisk2/_1009_qlora_ds_adddataset_cleanv1_int4/tree/None' for available files.

使用vllm部署报错：

KeyError: 'transformer.h.0.attn.c_attn.g_idx'

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?

No response

[Stark-zheng]

量化完直接推理：response, history = model.chat(tokenizer, query=test_data[0], history=None)

报错：AttributeError: 'QuantLinear' object has no attribute 'q4'