Open LeMoussel opened 7 months ago
Are you using the latest version of huggingface/transformers
? it seems your transformers
version is too old to support Mixtral.
Besides, as shown in readme you have to pass the modules_to_not_convert
argument for Mixtral for now because the gate layer is not quantized. I'll change this to auto-detection later.
quant_model = Exl2ForCausalLM.from_quantized("turboderp/Mixtral-8x7B-instruct-exl2",
revision="3.0bpw",
modules_to_not_convert=["gate"])
With this
I got this error
whereas if I use bartowski/dolphin-2.6-mistral-7b-dpo-laser-exl2, I have no error.
Any idea why this error occurs?