yxli2123 / LoftQ

MIT License
180 stars 15 forks source link

About the test result on gsm8k #33

Open lucasliunju opened 1 month ago

lucasliunju commented 1 month ago

Hi, I try to use this code to evaluate the performance of LoftQ:

python test_gsm8k.py \
    --model_name_or_path LoftQ/Llama-2-7b-hf-4bit-64rank \
    --batch_size 16

And I find that the final ACC is about 40.11%. But the reported result in the paper is 35.0%. So I would like to ask whether my test result is right.

Thanks