search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
7.41k stars 801 forks source link

calibration dataset #1100

Open Hukongtao opened 4 months ago

Hukongtao commented 4 months ago

I followed the steps provided in the tutorial to use awq-4bit quantization, but found that the accuracy was seriously lost after quantization. https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md#awq My model is the llama after SFT. Therefore, I want to know whether I need to use my own training set for calibration. When using awq-4bit result in dropped points, are there any parameters that can be adjusted?

ehuaa commented 2 months ago

+1 I have met the same problem with the accuracy drop after quantization, how can i debug with it or how to choose the calibration dataset? @Tracin @byshiue

cieske commented 1 month ago

+1 I also wonder if there is any reference for choosing an appropriate calibration dataset.