search

efeslab / Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
259 stars 21 forks source link

ppl on ptb #3

Closed MrDoghead closed 8 months ago

MrDoghead commented 8 months ago

When I ran your code, I found that the ppl score of the llama-7B model on the PTB dataset is greater than 20, whether quantized or not. And in another paper, the score is also different from yours. Why?

Screenshot 2024-01-12 at 19 10 49
happierpig commented 8 months ago

Hi @MrDoghead ,

Thanks for reporting the issue. I suppose the mismatch comes from the model you are using.

  1. The results we reported in our paper are aligned with SpQR.
  2. We evaluate widely used deca-llama. Huggy-llama may show different results.
  3. PTB is relatively small, therefore results are more erratic.
happierpig commented 8 months ago

After 24 hours