Closed gesanqiu closed 4 months ago
If you want to quantize a new model with the same architecture with supported model, you can just set the --net
directly:
This is an example command to quantize CodeLLama:
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/CodeLLama/CodeLLama-7b \
--epochs 20 --output_dir ./log/llama-7b-w3a16g128 \
--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc \
--net Llama-2-7b
Thanks for you reply, my question is how to add a new model which architecture is not supported yet.
You have to add new files int_{your model}_layer.py in models/
, refer to other files. There can be some modification about namings for register parameter(e.g. o_proj <-> out_proj, c_fc <-> fc2) and cpu offloading (e.g. model.transformer.h <-> model.decoder.layers) in main.py
and omniquant.py
.
You have to add new files int_{your model}_layer.py in
models/
, refer to other files. There can be some modification about namings for register parameter(e.g. o_proj <-> out_proj, c_fc <-> fc2) and cpu offloading (e.g. model.transformer.h <-> model.decoder.layers) inmain.py
andomniquant.py
.
Have you tried to add bloom models? I met some problems in issue #29 .
@Louym No, I haven't tried to add bloom models. I don't know the details of your implementation (and I'm not a contributor of this repo), and attached error has various reasons. Anyway, I hope you can solve this problem, and I hope my experience can be of help.
Thanks for your brilliant work, after explord the project for several days, I found that OmniQuant is portable for edge devices, like Jetson or phones. And wondering how can I add more models into OmniQuant, do you have any tutorials about this? Or maybe we can start from CodeLlama, since it has the similiar architecture with Llama-2, and Llama-2 is already supported. Also apologies in advance if this seems to be something obvious because I'm new in LLM field.