Closed ghost closed 10 months ago
--model
as your model path--net
as Llama-2-13b
For example:
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/YOUR/MODEL --eval_ppl \
--epochs 20 --output_dir /PATH/TO/log \
--wbits 3 --abits 16 --group_size 128 --lwc \
--net Llama-2-13b
@ChenMnZ what about --calib_dataset
? Do I need to adjust this argument? I finetuned this model on custom dataset with format:
{'instruction' : "",
"input": "",
"output": ""}
We found that current wikitext2
calibration dataset can quantize instruction-tuned models well.
Maybe you can try to add --aug_loss
, which benefit to some quantization.
Can you guide me on how to quantize a custom LLama2-13B model finetuned on an alpaca-liked dataset ? Thank you so much!