In awq_int4_group_128 mode, why is the weight converted into int8 after rounding to [-8,7]?

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

https://nvidia.github.io/TensorRT-LLM

Apache License 2.0

8.75k stars 1k forks source link

In awq_int4_group_128 mode, why is the weight converted into int8 after rounding to [-8,7]? #1120

Open brisker opened 9 months ago

brisker commented 9 months ago

When building AWQ_int4_group_128 engine, the code here seems to convert the weights which are already rounded to [-8,7] into int8, using the following two command defined [here] and here.

packer = torch.ops.trtllm.pack_int8_tensor_to_packed_int4
preprocessor = torch.ops.trtllm.preprocess_weights_for_mixed_gemm

int4_weight = preprocessor(packer(qweight_int8.cpu()), torch.quint4x2)

Isn't this INT4 weight quantization? Why is int8 engaged here? This is confusing to me.

Tracin commented 9 months ago

Looks like we pack 2 int4 weights into one int8 number.

hello-11 commented 1 week ago

@brisker Do you still have the question? If not, we will close it soon.