Open condy0919 opened 6 months ago
The params_dict
constructed from dict(self.named_parameters())
only has "SqueezeLLM"ed parameters.
So there is no such model.layers.0.self_attn.qkv_proj.weight
anymore.
I used the try catch continue to jump to the weights
I used the try catch continue to jump to the weights
Is the accuracy of the tasks affected?
Facing this issue with --quantization=gguf
Same issue with MistralLarge BitsAndBytes
Your current environment
🐛 Describe the bug
The last error is
This issue is simillar to #4013, but it still failed with latest vllm.