Closed gesanqiu closed 11 months ago
If you want to run your own really quantized model, you should first save the model through --save_dir
and --real_quant
. For example:
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b \
--epochs 0 --output_dir temp \
--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc \
--resume PATH/TO/OMNI_PARAMETERS
--real_quant --save_dir PATH/TO/SAVE/MODEL
Then, you can run the quantized model by replacing the following variates in runing_falcon180b_on_single_a100_80g.ipynb
with the corresponding values:
#model_path = './pre_quantized_models/falcon-180b-omniquant-w3a16g512'
#wbits = 3
#group_size = 512
model_path = PATH/TO/SAVE/MODEL
wbits = 3
group_size = 128
You can set group_size
as -1
to disable group quantization, and if you want to test in more datasets, maybe you can refer to the --tasks
. For examle:
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b \
--epochs 0 --output_dir temp \
--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc \
--tasks piqa,arc_easy,arc_challenge,boolq,hellaswag,winogrande
As for NAN, it may be caused by mixed-precision training, you can use --deactive_amp
to use full-precision training.
Thanks for your detailed reply, I found that the NAN is my model's problem so I fallback to official llama-2-7b to explore OmniQuant.
I tried to run a quantized llama-2-7b-chat-w4a16g128 model with runing_falcon180b_on_single_a100_80g.ipynb
but it failed with following logs:
Traceback (most recent call last):
File "/workdir/test/test_omniquant.py", line 35, in <module>
layers = model.transformer.h
File "/root/anaconda3/envs/omniquant/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LlamaForCausalLM' object has no attribute 'transformer'
Except this problem, I think LlamaForCausalLM
also doesn't include FalconLinear
instance, so it may failed in get_named_linears(layer)
?
I replaced the code like follow and it worked:
from torch import nn
def get_named_linears(module):
return {name: m for name, m in module.named_modules() if isinstance(m, nn.Linear)}
# layers = model.transformer.h
layers = model.model.layers
I trained some llama structure like models, and use following command quatize my model:
it failed with following outpus:
In which case will produce a NULL Loss object? BTW, this command works on llama-7b well.
Besides PPL evaluation, I also need to do subject/object evaluaiton on some dataset. I know there is a
runing_falcon180b_on_single_a100_80g.ipynb
, which shown how to run a quantized Falcon-180b, but it seems I have to learn AutoGPTQ first before I can know how to load quantized weight of a model?I convert the quantized
llama-7b-w4a16.pth
to hf format with https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py. I tried replaced theFalconLinear
tonn.Linear
, not sure if it's correct. And I wander what the group size is, if I didn't use group size in quantize phase, what value should I set inQuantLinear
?Are there any examples or docs can help me run a llama model quickly?