How to support activation 4bit quantization?

[Ther-nullptr]

As mentioned in title.

[dacorvo]

Why 4-bit activations are not supported in quanto ?

Activations are quantized dynamically based on the recording of scales during calibration (unlike weights that are quantized statically), adding an extra cost to the inference. To make it worth it, we need to benefit from an accelerated matmul with the quantized weights. Unfortunately, at this stage the only accelerated operations available are for scalar quantization scales, that give terrible results with 4-bit weights (you need group-wise scales to preserve accuracy).

How could you still use 4-bit activations ?

You would need to modify some code here and there:

• add support for QBitsTensor in activation tensors,
• add support for group-wise input and output scales in quantized modules,
• add support for group-wise scales calibration.