search

bytedance / ABQ-LLM

An acceleration library that supports arbitrary bit-width combinatorial quantization operations
Apache License 2.0
226 stars 25 forks source link

Question about W4A16 Benchmark #17

Open RanchiZhao opened 3 weeks ago

RanchiZhao commented 3 weeks ago

I benchmarked the downstream performance of the ABQ W4A16 (per-channel, symmetric) quantized version of the MiniCPM-1B-sft-llama-format model. I used a custom dialogue dataset for calibration and modified the input_ids and labels logic in datautils.py. However, the performance did not match that of AutoGPTQ (with act_order). Do you have any suggestions?

zengchao0424 commented 2 weeks ago

Hi, for the W4A16 configuration, most mainstream quantization algorithms can achieve good results. If using a per-channel setting for W4A16, you can choose any quantization algorithm you prefer. However, in more challenging settings like W3 or W2, AWQ/GPTQ will experience significant accuracy drops under per-channel settings, where the advantages of ABQ will become more apparent.

RanchiZhao commented 2 weeks ago

Hi, for the W4A16 configuration, most mainstream quantization algorithms can achieve good results. If using a per-channel setting for W4A16, you can choose any quantization algorithm you prefer. However, in more challenging settings like W3 or W2, AWQ/GPTQ will experience significant accuracy drops under per-channel settings, where the advantages of ABQ will become more apparent.

Hi, The w4a16 per-channel quantized models perform quite well on multiple choice benchmarks. However, when we look at benchmarks that require longer outputs, like gsm8k or humaneval, the performance doesn't seem to be as good. We're interested in pushing the limits of the w4a16 per-channel quantization methods. In this case, it's still a challenge because we've set higher expectations for its performance.