Recommended Torch Quantization Library to Use -- Modelopt v.s. Pytorch-quantization

[YixuanSeanZhou]

Description

I have seen two quantization librares built by Nvidia: a TRT modelopt and a pytorch-quantization. What are the differences between the two libraries?

My use case is to do PTQ (and potentially QAT) on a pytorch model, export it to onnx and then covert it with TRT. In this case, what library should I choose to use?

Thanks,

[lix19937]

TRT modelopt include pytorch-quantization.

See https://nvidia.github.io/TensorRT-Model-Optimizer/guides/_pytorch_quantization.html

ModelOpt PyTorch quantization is refactored based on pytorch_quantization.

Key advantages offered by ModelOpt’s PyTorch quantization:

1 Support advanced quantization formats, e.g., Block-wise Int4 and FP8.

2 Native support for LLM models in Hugging Face and NeMo.

3 Advanced Quantization algorithms, e.g., SmoothQuant, AWQ.

4 Deployment support to ONNX and NVIDIA TensorRT.

Both tools you can use.

https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/index.html#

[YixuanSeanZhou]

Gotcha! Thank you for the clarification!