-
Hi, I'm using `BitsAndBytesConfig` on HF's Transformers library to quantize `facebook/opt-66B` model. But when I print the dtype of weights of varoius layers, all of them turn out to be of `int8`.
…
-
I transfer int8 model successfully, but get error when inference.
Error as flow:
self.model_trt.load_state_dict(torch.load(model_path))
File "/usr/local/lib64/python3.6/site-packages/torch/nn…
-
Hi, thanks for the great code.
It works well without any modification.
However, the result that I ran seems curious. My GPU is Tesla V100-SXM2-32GB and CPU is Intel(R) Xeon(R) Gold 5120 CPU @ 2.…
-
Are there any runnable demos of using Sparse-QAT/PTQ (2:4) to accelerate inference, such as applying PTQ to a 2:4 sparse LLaMA for inference acceleration? I am curious about the potential speedup rati…
-
Hi everyone,
I’m working on a project that involves deploying a YOLOv10 model on a mobile/edge device. To improve inference speed and reduce the model size, I want to convert my YOLOv10 model to Te…
-
Inspired by a recent back and forth with @gau-nernst we should add some quantized training recipes in AO for small models (600M param range)
Character.ai recently shared that they're working on qua…
-
Hi,
are you running float32 neural network in the chip?
do you chip support int8 NPU?
do you have the quantization tool to quantize model and then run it in the chip?
-
## Description
I generated calibration cache for Vision Transformer onnx model using EntropyCalibration2 method. When trying to generate engine file using cache file for INT8 precision using trte…
-
I tried quantized YOLOv3 on Volta GPU.
But, it didn't seem to be run on Tensor Core.
CUDNN documentation in 2.8.2 recommends to use "CUDNN_DATA_INT8x32 " for Tensor Core operations.
https://docs.nv…
-
- quantize (OK)
```bash
python3 -m modelopt.onnx.quantization --onnx_path encoder.onnx \
--quantize_mode int8 --output_path encoder-w8a8-int8.onnx
/root/anaconda3/envs/modelopt/lib/pyt…