search

efeslab / Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
259 stars 21 forks source link

The question about calib data #15

Closed ghost closed 5 months ago

ghost commented 5 months ago

Hi, I am wandering which dataset should be the calib dataset? I want to evaluate the quantized model on my own dataset. Which dataset to generate the calib data? my own dataset or other public dataset like wikitext?

happierpig commented 5 months ago

Hi @mxjyst ,

You can go with WikiText-2. The calib dataset should be different from the evaluation dataset to demonstrate the generality.

happierpig commented 5 months ago

By the way, we updated the codebase recently to support FP4 accuracy evaluation. You can try it with arg "--quant_type fp".

ghost commented 5 months ago

I'll try it later. Thanks a lot.