search

xhedit / quantkit

cli tool to quantize gguf, gptq, awq, hqq and exl2 models
MIT License
60 stars 4 forks source link

Mixtral #3

Closed Sneakr closed 4 months ago

Sneakr commented 5 months ago

Seems not work for mixtral.

File "/home/eyera/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/quantkit/convert.py", line 1417, in do_gguf_conversion

model = convert_model_names(model, params, False) #skip_unknown

File "/home/eyera/miniconda3/envs/unsloth_env/lib/python3.10/site-packages/quantkit/convert.py", line 1207, in convert_model_names

raise Exception(f"Unexpected tensor name: {name}. Use --skip-unknown to ignore it (e.g. LLaVA)")

Exception: Unexpected tensor name: model.layers.0.block_sparse_moe.gate.weight. Use --skip-unknown to ignore it (e.g. LLaVA)

xhedit commented 5 months ago

I pushed some updates that should make this work but the resulting file sizes seem off, so I probably also need to update llama-cpp-conv too.

xhedit commented 4 months ago

Mixtral tested out OK with an updated llama-cpp-conv, so I am closing this.