-
# 设备
unbuntu20
nvidia 1080ti
cuda 11.7
ternsorrt 8.6
paddleslim 2.6
paddle 2.6
# 复现过程
1. 下载链接提供yolov5s.onnx, 跑pytorch_yolo_series样例,得到量化后模型
2. onnx推理预测,fp32 和 fp16结果均正常, int8没有框;
3. 运行时…
GDbbq updated
7 months ago
-
I would like to quantize my model to INT8 precision and then compile it using torch_tensorrt.
Unfortunately, it is [transformer based vision model](https://github.com/mit-han-lab/efficientvit/blob/ma…
-
## Description
yolov5s base pytorch-quantization
reference https://github.com/maggiez0138/yolov5_quant_sample
onnx->fp16 3ms
qat->onnx->int8 4ms
why? please tell me,thanks.
[onnx fil…
-
My GPU
`GPU GTX 1070 `
I developing on Docker
The versions I'm working with
```
deepstream-app version 6.3.0
DeepStreamSDK 6.3.0
CUDA Driver Version: 12.1
CUDA Runtime Version: 12.1
Ten…
-
## official resnet50
* https://forums.developer.nvidia.com/t/jetson-agx-xavier-deep-learning-inference-benchmarks/67844
* https://developer.nvidia.com/embedded/jetson-agx-xavier-dl-inference-benchma…
-
### Describe the issue
Hi, I am using ONNX Runtime with TensorRT Execution Provider for a quantized model (YOLO-NAS). While TensorRT cli (trtexec.exe) successfully build the engine from onnx model, t…
-
I have `INT8` quantized a `BERT` model for binary text classification and am only getting a marginal improvement in speed over `FP16`.
I am using the `transformer-deploy` library that utilizes Tens…
-
https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/README.md#int8-kv-cache
When using int8 kv-cache, you will encounter the above warning. Will this affect accuracy?
-
Hello!
Does TensorRT-LLM supports Medusa with Mixtral 8x7B?
My understanding is that right now the Medusa [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/medusa/c…
-
/code/tensorrt_llm# python examples/llama/convert_checkpoint.py --model_dir /code/tensorrt_llm/Mixtral-8x7B-Instruct-v0.1/ --dtype float16 --output_dir /code/tensorrt_llm/examples/Mixtral-8x7B-Instruc…