Open ChengZhang-98 opened 5 months ago
Dear author, may I ask how to evaluate the W6A6 opt model using the provided ChenMnZ/OmniQuant, act_shifts, and act_scales ?
act_shifts
act_scales
Here is my command
python main.py --model $PATH_TO_MY_OPT_CHECKPOINT \ --epochs 0 \ --output_dir $LOG_DIR \ --wbits 6 --abits 6 --lwc --let \ --tasks lambada_openai \ --resume ./OmniQuant/opt-6.7b-w6a6.pth # this is the ckpt downloaded from the huggingface repo
I got the following error raise in https://github.com/OpenGVLab/OmniQuant/blob/ec4adedf42b17cf30a6d5faf0270e2036d2579f3/quantize/utils.py#L106
AttributeError: 'QuantOPTDecoderLayer' object has no attribute 'input_layernorm'. Did you mean: 'final_layer_norm'?
If I evaluate W6A6 llama-7b using the following command, the lambada accuracy is 0, though the perplexity matches the value reported in the paper.
❓ May I ask if I missed out something in the following command line?
python main.py --model $PATH_TO_MY_LLAMA_CHECKPOINT \ --epochs 0 \ --output_dir $LOG_DIR \ --wbits 6 --abits 6 --lwc --let \ --tasks lambada_openai \ --resume ./OmniQuant/llama-7b-w6a6.pth # this is the ckpt downloaded from the huggingface repo
Here is the metrics I copied from the output
{'config': {'bootstrap_iters': 100000, 'description_dict': None, 'limit': None, 'model': <models.LMClass.LMClass object at 0x7fc3624658a0>, 'model_args': None, 'num_fewshot': 0}, 'results': {'arc_challenge': {'acc': 0.38822525597269625, 'acc_norm': 0.4112627986348123, 'acc_norm_stderr': 0.01437944106852208, 'acc_stderr': 0.014241614207414037}, 'arc_easy': {'acc': 0.6637205387205387, 'acc_norm': 0.5197811447811448, 'acc_norm_stderr': 0.010251751199542736, 'acc_stderr': 0.009694178072725202}, 'boolq': {'acc': 0.728440366972477, 'acc_stderr': 0.00777897092960314}, 'lambada_openai': {'acc': 0.0, 'acc_stderr': 0.0, 'ppl': 2654605.7843538206, 'ppl_stderr': 129661.85146222409}, 'openbookqa': {'acc': 0.272, 'acc_norm': 0.418, 'acc_norm_stderr': 0.022080014812228134, 'acc_stderr': 0.01992048320956608}, 'piqa': {'acc': 0.7671381936887922, 'acc_norm': 0.764417845484222, 'acc_norm_stderr': 0.009901067586473888, 'acc_stderr': 0.009861236071080746}}, 'versions': {'arc_challenge': 0, 'arc_easy': 0, 'boolq': 1, 'lambada_openai': 0, 'openbookqa': 0, 'piqa': 0}}
I have fixed the bug about OPT models in the latest code.
Dear author, may I ask how to evaluate the W6A6 opt model using the provided ChenMnZ/OmniQuant,
act_shifts
, andact_scales
?Here is my command
I got the following error raise in https://github.com/OpenGVLab/OmniQuant/blob/ec4adedf42b17cf30a6d5faf0270e2036d2579f3/quantize/utils.py#L106
If I evaluate W6A6 llama-7b using the following command, the lambada accuracy is 0, though the perplexity matches the value reported in the paper.
❓ May I ask if I missed out something in the following command line?
Here is the metrics I copied from the output