Using cv2.dnn to use our trained model

prp-e commented 3 years ago

How can I use `cv2.dnn` to use model I trained using YOLOv5?

I have a super old code (which is for YOLOv3 I believe) which looks like this:

import cv2 
import uuid

image = cv2.imread(f'girl-and-car.jpg')

labels = []
with open('coco.names', 'rt') as f:
    labels = f.read().rstrip('\n').split('\n')

weights = 'frozen_inference_graph.pb'
config = 'ssd_mobilenet_v3_large_coco_2020_01_14.pbtxt'

net = cv2.dnn_DetectionModel(weights, config)
net.setInputSize(320, 320)
net.setInputScale(1.0 / 127.5)
net.setInputMean((127.5, 127.5, 127.5))
net.setInputSwapRB(True)

ids, confidences, binding_box = net.detect(image, confThreshold = 0.5)

for id, confidence, box in zip(ids.flatten(), confidences.flatten(), binding_box):
    cv2.rectangle(image, box, color=(0,0,255), thickness=2)
    cv2.putText(image, labels[id-1].upper(), (box[0]+10, box[1]+30), cv2.FONT_HERSHEY_COMPLEX, 1, (0,0,255), 2)

cv2.imshow("Result", image)
cv2.waitKey(0)

cv2.imwrite(f'outputs/{str(uuid.uuid4())}.jpg', image)

And using YOLOv5 I did this:

model = torch.hub.load('ultralytics/yolov5', 'custom', path='yolov5/runs/train/exp3/weights/last.pt', force_reload=True)

Is it possible to feed this weights/last.pt to DNN? I run my code on CPU and using pytroch for detection, makes my code slow as hell. I'd be thankful if you help me get it to work with cv2.dnn method.

github-actions[bot] commented 3 years ago

👋 Hello @prp-e, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Google Colab and Kaggle notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

prp-e commented 3 years ago

Apparently it is possible to read from ONNX, So I used export.py to generate a decent ONNX model.

prp-e commented 3 years ago

I made a simple onnx export using:

!cd yolov5 && python3 export.py --weights runs/train/exp3/weights/last.pt --img 320 --batch 16

and this is what I got:

export: weights=runs/train/exp3/weights/last.pt, img_size=[320], batch_size=16, device=cpu, include=['torchscript', 'onnx', 'coreml'], half=False, inplace=False, train=False, optimize=False, dynamic=False, simplify=False, opset_version=12
YOLOv5 🚀 v5.0-287-gd204a61 torch 1.8.1 CPU

Fusing layers... 
Model Summary: 308 layers, 21098253 parameters, 0 gradients, 50.5 GFLOPs

PyTorch: starting from runs/train/exp3/weights/last.pt (42.5 MB)

TorchScript: starting export with torch 1.8.1...
/Users/prp-e/playground/yolov5_pytorch/yolov5/models/yolo.py:58: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.grid[i].shape[2:4] != x[i].shape[2:4] or self.onnx_dynamic:
TorchScript: export success, saved as runs/train/exp3/weights/last.torchscript.pt (84.8 MB)
ONNX: starting export with onnx 1.10.1...
ONNX: export success, saved as runs/train/exp3/weights/last.onnx (84.4 MB)
CoreML: export failure: No module named 'coremltools'

Export complete (30.94s). Visualize with https://github.com/lutzroeder/netron.

and I wrote this to test it out:

import cv2 
import numpy as np
import onnx 

net = cv2.dnn.readNetFromONNX("yolov5/runs/train/exp3/weights/last.onnx")

image = cv2.imread("data/images/hotdogs-25.jpg")
image = cv2.resize(image, (320, 320))

blob = cv2.dnn.blobFromImage(image, 1.0 , (320, 320))
net.setInput(blob)

results = net.forward()
biggest_pred_index = np.array(results)[0].argmax()
print(biggest_pred_index)

cv2.imshow("image", image)
cv2.waitKey(0)

but when I run test code, I get:

[ERROR:0] global /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-xxsyexfp/opencv/modules/dnn/src/onnx/onnx_importer.cpp (2127) handleNode DNN/ONNX: ERROR during processing node with 3 inputs and 1 outputs: [Range]:(528)
Traceback (most recent call last):
  File "onnx_test.py", line 5, in <module>
    net = cv2.dnn.readNetFromONNX("yolov5/runs/train/exp3/weights/last.onnx")
cv2.error: OpenCV(4.5.3) /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-xxsyexfp/opencv/modules/dnn/src/onnx/onnx_importer.cpp:2146: error: (-2:Unspecified error) in function 'handleNode'
> Node [Range]:(528) parse error: OpenCV(4.5.3) /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-xxsyexfp/opencv/modules/dnn/src/dnn.cpp:621: error: (-2:Unspecified error) Can't create layer "528" of type "Range" in function 'getLayerInstance'

glenn-jocher commented 3 years ago

@prp-e your code is out of date. You should update and then run ONNX inference with detect.py:

python export.py --weights yolov5s.pt --include onnx --dynamic
python detect.py --weights yolov5s.onnx

For cv2 errors, I would raise those directly on the cv2 repository as they are out of our control.

prp-e commented 3 years ago

Thanks. I've updated my base, I'll check it out.

glenn-jocher commented 3 years ago

@prp-e good news 😃! Your original issue may now be fixed ✅ in PR #4833 by @SamFC10. This PR implements architecture updates to allow for ONNX-exported YOLOv5 models to be used with OpenCV DNN.

To receive this update:

Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
PyTorch Hub – Force-reload with model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
Notebooks – View updated notebooks
Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

willyd commented 3 years ago

@glenn-jocher @SamFC10 I have the same issue. I am getting this error:

cv2.error: OpenCV(4.5.4) /home/gdumont/dev/4elements/opencv-4.5.4/modules/dnn/src/onnx/onnx_importer.cpp:739: error: (-2:Unspecified error) in function 'handleNode'
> Node [Range]:(355) parse error: OpenCV(4.5.4) /home/gdumont/dev/4elements/opencv-4.5.4/modules/dnn/src/dnn.cpp:615: error: (-2:Unspecified error) Can't create layer "355" of type "Range" in function 'getLayerInstance'
>

I have tried all export commands I have seen on Github without success. Is there anything special I need to do in order to export a model without a Range operator?

willyd commented 3 years ago

From what I understand we need to export the models without the --dynamic option.

jebastin-nadar commented 3 years ago

@willyd Yes, onnx export with --dynamic option produces a Range layer which is not supported in OpenCV.

Code for testing

import numpy as np
import cv2

inp = np.random.rand(1, 3, 640, 640).astype(np.float32)
net = cv2.dnn.readNetFromONNX('yolov5s.onnx')
net.setInput(inp)
out = net.forward()
print(out.shape)

onnx model without --dynamic (1, 25200, 85)

onnx model with --dynamic

cv2.error: OpenCV(4.5.4-dev) /home/jebastin/opencv/opencv/modules/dnn/src/onnx/onnx_importer.cpp:765: error: (-2:Unspecified error) in function 'handleNode'
Node [Range]:(354) parse error: OpenCV(4.5.4-dev) /home/jebastin/opencv/opencv/modules/dnn/src/dnn.cpp:615: error: (-2:Unspecified error) Can't create layer "354" of type "Range" in function 'getLayerInstance'

There is already a feature request for this layer https://github.com/opencv/opencv/issues/20324. Meanwhile you can use onnx models without --dynamic option.

willyd commented 3 years ago

Thanks @SamFC10 for the clarification. Somehow I thought all examples add the --dynamic switch. But they don't. My bad.

prp-e commented 2 years ago

My export code is:

!cd yolov5 && python3 export.py --weights runs/train/exp8/weights/last.pt --include onnx --dynamic

and when I try to open it with cv2.dnn, I get this:

error                                     Traceback (most recent call last)
<ipython-input-9-00f4dbda0784> in <module>
----> 1 net = cv2.dnn.readNetFromONNX('pedestrians.onnx')

error: OpenCV(4.5.4-dev) /Users/runner/work/opencv-python/opencv-python/opencv/modules/dnn/src/onnx/onnx_importer.cpp:739: error: (-2:Unspecified error) in function 'handleNode'
> Node [Range]:(405) parse error: OpenCV(4.5.4-dev) /Users/runner/work/opencv-python/opencv-python/opencv/modules/dnn/src/dnn.cpp:615: error: (-2:Unspecified error) Can't create layer "405" of type "Range" in function 'getLayerInstance'
>

what's the problem now?

glenn-jocher commented 2 years ago

@prp-e as explained in this issue above --dynamic does not support DNN inference. Workaround is to export with default settings.

willyd commented 2 years ago

@prp-e @glenn-jocher is right. Remove the dynamic option and it should work. If you want to use dynamic image sizes you can export with --dynamic and use onnxruntime instead of OpenCV for inference.

prp-e commented 2 years ago

Thanks, my problem with loading ONNX solved. Now, I wrote this piece of code:

inp = cv2.imread('pedestrians.jpg')
net = cv2.dnn.readNetFromONNX('pedestrians_new.onnx')

inp = cv2.resize(inp, (640, 640))

net.setInput(inp)
out = net.forward()

and I get this result:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-15-9d8f58eac663> in <module>
----> 1 out = net.forward()

error: OpenCV(4.5.4-dev) /Users/runner/work/opencv-python/opencv-python/opencv/modules/dnn/src/layers/slice_layer.cpp:202: error: (-215:Assertion failed) sliceRanges_rw[i].size() <= inpShape.size() in function 'getMemoryShapes'

jebastin-nadar commented 2 years ago

@prp-e Input shape is incorrect. Model expects inputs of size (1, 3, 640, 640), you are passing an input of size (640, 640, 3). Make sure your input pre-processing steps are correct (use detect.py as reference)

prp-e commented 2 years ago

Thanks @SamFC10. I couldn't find the reshaping process from detect.py. May you help me a little bit?

jebastin-nadar commented 2 years ago

@prp-e Check if this works

import numpy as np
import cv2

inp = cv2.imread('pedestrians.jpg')
blob = cv2.dnn.blobFromImage(inp, 1.0/255, (640, 640), swapRB=True)

net = cv2.dnn.readNetFromONNX('pedestrians_new.onnx')
net.setInput(blob)
out = net.forward()

prp-e commented 2 years ago

Thanks @SamFC10, it worked perfectly. I also have a question about bounding boxes, etc. How can I add them to my inp picture and show them?

slience-ops commented 2 years ago

@prp-e How to solve the " loading ONNX"?
error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20220524::ONNXImporter::handleNode'

Node [Identity@ai.onnx]:(onnx_node!Identity_0) parse error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\dnn\src\layer.cpp:246: error: (-215:Assertion failed) inputs.size() in function 'cv::dnn::dnn4_v20220524::Layer::getMemoryShapes'

ultralytics / yolov5