-
I quantized a cnn to an int8 one and measured the average inference speed, the latency is 2x slower than the original fp32 model. Then I used the profile tools and printed out latency per layer. It tu…
-
I have experimented multiple models with ARM-NN on Cortex A53(mostly int8 quantized models with latency < 200ms). And I found XNNPACK generally gives a better latency result than ARM-NN. So I am tryin…
-
### Description of the bug:
I'm trying to understand how to pass quantization config to convert function of ai_edge_torch. But whenever I pass the representative dataset, it causes the colab noteboo…
-
### Problem Description
Seeing GPU fault when running the onnxruntime-inference-examples script using reduced layer bert models during benchmarking.
It appears quantization/calibration steps work …
-
### Describe the issue
I do a qat quantization on a cnn model, when a export it to onnx model, and got a slower inference than torchscript qat model.
the result is
torchscript: 4.798517942428589 …
-
### Before Asking
- [X] I have read the [README](https://github.com/meituan/YOLOv6/blob/main/README.md) carefully. 我已经仔细阅读了README上的操作指引。
- [X] I want to train my custom dataset, and I have read the …
-
**Describe the bug**
Trying to use DeepSpeed Inference with int8 does not work for GPTJ. I get created an issue that has more details on the DeepSpeed MII repo, but due to the nature of the issue, I…
-
# Bug Report
### Describe the bug
I use quantize_static and convert_float_to_float16 functions in onnx to convert fp32 model to fp16 + int8 model. The fp16 model can inference through onnxruntime …
Ai-ZL updated
4 hours ago
-
I correctly converted yolov7 from float32 to an int8 HEF and inference on hailo-8, but when I used the benchmark, I found that my model's FPS (28FPS) is only half of the official benchmark (46FPS).The…
-
Is it currently possible to use the int8 type for training and inference with nanoGPT? I only see different floating-point options in train.py. What might it take to add such support if it’s currentl…