Loading onnx model with torch.hub.load

ghost commented 2 years ago

Search before asking

[x] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello, How can i load onnx model properly ? I'm trying to load an onnx model with torch hub load, but I get this error:

Downloading: "https://github.com/ultralytics/yolov5/archive/master.zip" to C:\Users\ItsRa/.cache\torch\hub\master.zip
YOLOv5  2022-5-25 Python-3.8.0 torch-1.11.0+cu113 CUDA:0 (NVIDIA GeForce RTX 2060, 6144MiB)

Loading brain.onnx for ONNX Runtime inference...
Adding AutoShape...
Traceback (most recent call last):
  File "C:\Users\ItsRa\Desktop\Aimly\Brain.py", line 187, in <module>
    Aimly.Start()
  File "C:\Users\ItsRa\Desktop\Aimly\Brain.py", line 139, in Start
    DetectionResults = self.model(CurrentFrame)
  File "C:\Users\ItsRa\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(input, **kwargs)
  File "C:\Users\ItsRa\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(args, kwargs)
  File "C:\Users\ItsRa/.cache\torch\hub\ultralytics_yolov5_master\models\common.py", line 602, in forward
    y = self.model(x, augment, profile)  # forward
  File "C:\Users\ItsRa\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, kwargs)
  File "C:\Users\ItsRa/.cache\torch\hub\ultralytics_yolov5_master\models\common.py", line 454, in forward
    y = self.session.run([self.session.get_outputs()[0].name], {self.session.get_inputs()[0].name: im})[0]
  File "C:\Users\ItsRa\AppData\Local\Programs\Python\Python38\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 192, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: images for the following indices
 index: 0 Got: 1 Expected: 16
 index: 2 Got: 640 Expected: 416
 index: 3 Got: 640 Expected: 416
 Please fix either the inputs or the model.

I'm trying to load it with

    def __init__(self, BoxSize = 416, CollectDataMode = False, MouseDelay = 0.0001, RunDebugMode = False):
        self.BoxSize = BoxSize
        self.model = torch.hub.load('ultralytics/yolov5', 'custom', path='brain.onnx', force_reload = True)
        self.model.conf = 0.4 # Base confidence TargetLockThreshold (or base detection (0-1)
        self.model.iou = 0.4 # NMS IoU (0-1)
        self.CollectDataMode = CollectDataMode
        self.MouseDelay = MouseDelay
        self.RunDebugMode = RunDebugMode

Additional

No response

github-actions[bot] commented 2 years ago

👋 Hello @iHafez, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Google Colab and Kaggle notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on macOS, Windows, and Ubuntu every 24 hours and on every commit.

glenn-jocher commented 2 years ago

@iHafez 👋 hi, thanks for letting us know about this possible problem with YOLOv5 🚀. ONNX models run correctly with PyTorch Hub, I just tested myself now.

We've created a few short guidelines below to help users provide what we need in order to start investigating a possible problem.

How to create a Minimal, Reproducible Example

When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:

✅ Minimal – Use as little code as possible to produce the problem
✅ Complete – Provide all parts someone else needs to reproduce the problem
✅ Reproducible – Test the code you're about to provide to make sure it reproduces the problem

For Ultralytics to provide assistance your code should also be:

✅ Current – Verify that your code is up-to-date with GitHub master, and if necessary git pull or git clone a new copy to ensure your problem has not already been solved in master.
✅ Unmodified – Your problem must be reproducible using official YOLOv5 code without changes. Ultralytics does not provide support for custom code ⚠️.

If you believe your problem meets all the above criteria, please close this issue and raise a new one using the 🐛 Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem.

Thank you! 😃

matrs commented 2 years ago

@glenn-jocher , your example works because you are passing a list with only 1 image, but this still fails if you pass a list with two or more images, contrary to how model works with the default model format.

glenn-jocher commented 2 years ago

@matrs if you want dynamic batch sizing for ONNX models you must pass --dynamic during export.

matrs commented 2 years ago

@glenn-jocher, thank you!

glenn-jocher commented 11 months ago

@matrs you're welcome! If you have any other questions or need further assistance, feel free to ask. Good luck with your project!

CD-SG commented 10 months ago

I can't get it to work when the onnx model was exported with --half

glenn-jocher commented 10 months ago

@CD-SG Currently, dynamic batch size is incompatible with models exported with '--half'. It's a known limitation, and we're working on addressing this in the future. Thank you for bringing it to our attention.

Lightn1ng0 commented 9 months ago

The actual problem I am having is loading the half precision onnx with torch.hub.load, and using it for predictions: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(float16)) , expected: (tensor(float)) It loads the model that was exported with half precision, but when predicting it errrors out like this.

The solution to this that I found was loading the model on onnxruntime directly instead of using torch hub What also doesnt work is exporting with the --half flag on cpu or amd, is there any possibility of exporting to a half model with cpu/amd? Or is it nvidia only? Since the half precision onnx models do work on cpu/amd I think its weird that they can't export it and you need someone with an nvidia card to export it with --half for you.

glenn-jocher commented 9 months ago

@Lightn1ng0 The error you're encountering is due to the data type mismatch between the model expecting float16 inputs and the inputs being provided as float32. ONNX Runtime currently requires inputs to be in the same data type as the model expects.

Regarding the --half flag, it is primarily used for GPUs, especially NVIDIA GPUs that support half-precision (float16) operations, which can lead to faster inference times and reduced model size. Exporting a model in half-precision on a CPU or AMD GPU may not be supported because these platforms typically do not perform operations in half-precision, and the PyTorch export functionality may be limited by the underlying hardware capabilities.

If you need to run half-precision models on hardware that doesn't support exporting in half-precision, you would indeed need access to an NVIDIA GPU to perform the export. Once exported, the ONNX model in half-precision can be loaded and run on other platforms that support ONNX Runtime and half-precision inference, even if they don't support half-precision training or exporting.

For now, if you need to use half-precision models on platforms without NVIDIA GPUs, your workaround of using ONNX Runtime directly is a valid approach. Make sure to convert your input data to float16 before passing it to the model for inference.

We appreciate your feedback and understand that these limitations can be inconvenient. We'll take note of this for future updates to the export functionality. Thank you for your patience and understanding.

ultralytics / yolov5