-
I wonder whether you can compared with SmoothQuant, it is a great quantization method that many people tend to use.
-
### Proposal to improve performance
I am using SmoothQuant to quantize Llama2-7b, but each sample takes 1 hour. Is this speed normal, and are there any ways to improve it?
![image](https://github.co…
-
**Describe the bug**
When using a SmoothQuantModifier and cpu offloading there is a conflict of tensors not being on the right device.
**Expected behavior**
cpu offloading should work w/ SmoothQu…
-
Hi, guys, Im wondering would smoothquant be supported in the future for int8 *onnx* quant?
Mainly for VIT-like model and LLM.
-
Hi,
I'm having issue when trying to convert starcoder2-3b with smoothquant to trtllm.
I'm running on a100-40gi.
This is my commad:
`python tensorrt_llm/examples/gpt/convert_checkpoint.py --mod…
-
Smoothquant implements input-weight equalization, currently the [implementation](https://github.com/pytorch/ao/blob/main/torchao/quantization/smoothquant.py) in torchao is using module swap, but it ca…
-
reference: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/quantization-in-TRT-LLM.md#performance
![image](https://github.com/user-attachments/assets/1bb20225-3eb2-4641-b5ba-f027…
-
Hi, I'm having trouble with the export_int8_model.py code results and would like to ask a question.
The model in the huggingface was fine, but I'm wondering what is causing the size issue when app…
-
-
### System Info
GPU Type: A6000
### Who can help?
_No response_
### Information
- [ ] The official example scripts
- [ ] My own modified scripts
### Tasks
- [ ] An officially sup…