search

facebookresearch / SpinQuant

Code repo for the paper "SpinQuant LLM quantization with learned rotations"
Other
152 stars 13 forks source link

Question about reproduction #10

Open kiucho opened 2 months ago

kiucho commented 2 months ago

Thanks for sharing great research.

I tried to reproduce the results of paper and got next problem. I tested Llama2-7b 4-16-16 (RTN) with 10_optimize_rotation.sh and got wikitext-2 ppl 5.5, which is exact same as paper reported.

BUT, when I tested zero-shot task with lm_eval==0.4.3, The results is as follows. ARC-e ARC-c PIAQ HellaSwag Winogrande
Paper 72.2 48.6 78.2 74.2 67.9
Mine 70.6 42.7 77.7 74.6 67.3

I think there's discrepancy in zero-shot results. Only difference is that paper results are tested on 8 x A100 and I tested on 8 x A6000. If the script 10_optimize_rotation.sh is not the exact script for repoducing, can I get exact script? Thanks.

BrotherHappy commented 2 months ago
+1 I use the official weight of Llama2-7b-w16-a4-kv4, and use the "ptq.py" to quantize it. The results are: taks ARC_easy ARC_challenge
paper 72.6 47.5
mine 70.7 41.8