search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.01k stars 880 forks source link

How to quantize customed models, such as LVM? #1945

Open XA23i opened 1 month ago

XA23i commented 1 month ago

Accodring to https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/quantization, Can I define my model and calibration process and then simply use modelopt.torch.quantization.quantize() ?

QiJune commented 1 month ago

@Tracin could you please have a look? Thanks

Tracin commented 1 month ago

I am not sure. Can we do that? @RalphMao

XA23i commented 1 month ago

By the way, what else can we do to quantize pytorch models besides modelopt.torch.quantization.quantize() function?

RalphMao commented 1 month ago

Yes we can quantize VLM, the only difference is that you need to prepare your calibration dataset to include both visual tokens and language tokens.

RalphMao commented 1 month ago

According to our experience, calibration VLM with language token only also gives decent results

github-actions[bot] commented 1 week ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."