Open XA23i opened 1 month ago
@Tracin could you please have a look? Thanks
I am not sure. Can we do that? @RalphMao
By the way, what else can we do to quantize pytorch models besides modelopt.torch.quantization.quantize() function?
Yes we can quantize VLM, the only difference is that you need to prepare your calibration dataset to include both visual tokens and language tokens.
According to our experience, calibration VLM with language token only also gives decent results
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."
Accodring to https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/quantization, Can I define my model and calibration process and then simply use modelopt.torch.quantization.quantize() ?