Is the Table.3 accuracy tested with dequantized weights, or tested on real accelerated quantized kernels?

mit-han-lab / qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Apache License 2.0

375 stars 15 forks source link

Is the Table.3 accuracy tested with dequantized weights, or tested on real accelerated quantized kernels? #17

Open vovoluck opened 2 months ago

vovoluck commented 2 months ago

For the zero-shot accuracy above, is the accuracy tested on dequantized weights, or tested on real accelerated quantized kernels?

If it is tested with dequantized weights, is there any accuracy difference between dequantized weights and real hardware quantized operations? Have you validated this? @synxlin

ys-2020 commented 2 months ago

Hi @vovoluck, thanks for your interests in QServe and QoQ.

The accuracy is tested with dequantized weights rather than the real accelerated kernels. We have verified that there is NO accuracy difference between dequantized weights and real hardware quantized operations.