[circle-quantizer] Support GPTQ

[01000-you]

What

• We propose supporting the GPTQ algorithm, a state-of-the-art post-training quantization (PTQ) method that has demonstrated robust performance, effectively compressing weights. Notably, GPTQ shows significant efficacy with quantization levels down to 4 bits and even 3 bits on a group-wise basis.

  Paper: https://arxiv.org/abs/2210.17323

Why

• The upcoming AI models support weight formats of up to 4 bits. To align with these changes, we need to integrate high-performance quantization algorithms such as GPTQ.

How

• The implementation will involve converting the Python code provided by the paper's authors into C++, and integrating it with the existing ONE codebase. The current processes in record-minmax and QuantDequant weightPass offer a procedural foundation suitable for the GPTQ algorithm which compute the Hessian via input and update weights to compensate for quantization errors.

Method

• The Hessian matrix is computed by averaging the values from 2X*X^T for every 100 input samples.
• Weight updating and quantization are executed row by row with compensation applied to each row to address quantization errors. This adjustment is carried out on a layer-by-layer basis.

Overview

[01000-you]

Experimental results

• The models below are vision classification models provided by PyTorch.
• We evaluated only using 4/8 bit weight quantization.
• Baseline : evaluated with FP32 model
• PTQ : original Post-training Quantization using QuantizeDequantizeWeightsPass

	Size			Accuracy
	NumParams	FP32 size	INT4 size	Baseline	PTQ-W8	GPTQ-W8	PTQ-W4	GPTQ-W4
DeiT	5,000,000	19.07MB	4.77MB	0.7202	0.7201	0.7202	0.6466	0.6918
EfficientFormer	12,290,000	46.88MB	11.72MB	0.8018	0.8002	0.8017	0.2023	0.77
ResNet18	11,689,512	44.59MB	11.15MB	0.6976	0.6974	0.6973	0.5821	0.6879
ResNet50	25,557,032	97.49MB	24.37MB	0.7615	0.7607	0.7611	0.5821	0.7557
RegNet400mf	4,344,144	16.57MB	4.14MB	0.7403	0.7395	0.7404	0.3613	0.7194
ResNeXt50	25,028,904	95.48MB	23.87MB	0.7761	0.7758	0.7763	0.6559	0.7686
Wide ResNet50	68,883,240	262.77MB	65.69MB	0.7848	0.7849	0.7847	0.7114	0.7801
Vgg16	138,357,544	527.79MB	131.95MB	0.7159	0.7156	0.7158	0.4644	0.6992
SqueezeNet	1,248,424	4.76MB	1.19MB	0.581	0.5796	0.5803	0.3335	0.5609
ShuffleNet_x0_5	1,366,792	5.21MB	1.30MB	0.6055	0.6021	0.6043	0.1033	0.3634

[jinevening]

The result in https://github.com/Samsung/ONE/issues/13480#issuecomment-2270215801 shows that GPTQ is effective in 4 bit weight quantization. For 8bits, the current PTQ works well for all benchmark models.

Do you have a plan to support 4 bit weight quantization?