Inquiry about Activation Quantization Strategy in Inference

[lucfisc]

Hi Team,

Great work, really interesting. I was wondering about an aspect in the paper. You are saying that you are using a "per token activation quantization ". Is it dynamic quantization or static at test time ?

In your test with MLC-LLM, you are only benchmarking Weight only quantization and the speed in INT2/ INT3 are worse than INT4. Is it because of constraint of MLC-LLM and that you are only looking for memory reduction ?

Thanks

[ChenMnZ]

Hi, thanks for your interest.

1. We utilize dynamic quantization for weight-activation quantization during test time.
2. The speed of weight-only quantization hinges on the executed kernels, making it a significant project. We selected MLC-LLM as it uniquely facilitates INT4/INT3/INT2 quantization. However, MLC-LLM's support for INT3/INT2 is currently suboptimal, particularly for INT3. Enhancements to INT3/INT2 quantization speed are in our future roadmap.

[lucfisc]

Thank you for the prompt answer. Closing the issue !