PPL results on wikitext/ptb/c4 are worse than the official result

IST-DASLab / gptq

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

https://arxiv.org/abs/2210.17323

Apache License 2.0

1.81k stars 145 forks source link

PPL results on wikitext/ptb/c4 are worse than the official result #34

Open xingyueye opened 1 year ago

xingyueye commented 1 year ago

Hi, I ran the bloom.py using fp16 to test the perplexity (PPL) of BLOOM on Wikitext-2, PTB, and C4 datasets. The results are 11.79 / 20.14 / 17.68, which is worse than the official results of 11.37/19.40/14.13.

efrantar commented 1 year ago

Hi, that's strange. I just tested a few models on my side with two different HF installs and the numbers still seem to reproduce. Could you provide some more details: what command are you running exactly, is this happening for all models or just a specific one, what are your huggingface, datasets and Python versions, etc.?

xingyueye commented 1 year ago

@efrantar My testing command is python bloom.py path/bloom-7b1/ c4 --wbits 16. I have tested several models and the results differ from the official results. One strange thing is that the difference increases along with the model size. Some of my env versions like that,

huggingface-hub          0.13.4
datasets                 2.10.0
transformers             4.28.1
torch                    2.0.0+cu117