onnx model

[never-to-never]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Why is the ONNX model much slower than PT model in reasoning? Is there any way to improve the reasoning speed of the ONNX model?

Additional

No response

[glenn-jocher]

@never-to-never ONNX models profile to similar or better speeds in our benchmarks:

Colab++ V100 High-RAM GPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=0, half=False
Checking setup...
YOLOv5 🚀 v6.1-39-gab2b1c0 torch 1.11.0+cu113 CUDA:0 (Tesla V100-SXM2-16GB, 16160MiB)
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 50.9/166.8 GB disk)

Benchmarks complete (488.23s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch      0.462296             8.556671
1             TorchScript      0.462296             5.429171
2                    ONNX      0.462296            13.180766
3                OpenVINO      0.462296            73.727725
4                TensorRT      0.462280             1.643648
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel           NaN                  NaN
7     TensorFlow GraphDef           NaN                  NaN
8         TensorFlow Lite           NaN                  NaN
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Colab++ V100 High-RAM CPU Results

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=cpu, half=False
Checking setup...
YOLOv5 🚀 v6.1-39-gab2b1c0 torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 42.3/166.8 GB disk)

Benchmarks complete (472.19s)
                   Format  mAP@0.5:0.95  Inference time (ms)
0                 PyTorch      0.462296           109.502023
1             TorchScript      0.462296           141.845495
2                    ONNX      0.462296            65.344190
3                OpenVINO      0.462296            68.984546
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.462296           119.991329
7     TensorFlow GraphDef      0.462296           119.357180
8         TensorFlow Lite      0.462334           224.775610
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

[github-actions[bot]]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!