spmallick / learnopencv

Learn OpenCV : C++ and Python Examples
https://www.learnopencv.com/
20.92k stars 11.55k forks source link

Onnx Model question #677

Open ghost opened 2 years ago

ghost commented 2 years ago

Hello Satya Mallick, my question is how you converted the Pytorch models to .onnx models. I assume https://github.com/ultralytics/yolov5 is being used (export.py, command : python export.py --weights model.pt --img 640). I have no problems loading the models (Opencv: 4.5.4) provided in this repo but my own trained model (Yolov5s) converted to .onnx wont detect anything neither even load correctly. My dataset is tested in Yolov4, Yolov4-tiny and its really good. I tried both readNet() and readfromONNX() and both load my model successfully but nothing more. Maybe important to mention is that my Inputsize is 256x256 in the Inference programm and all the images i trained on it. Could it somehow be that only 640x640 for input size is supported or that there are any known issues for lower input sizes?

brmarkus commented 2 years ago

You might want to create an issue under ´https://github.com/ultralytics/yolov5´ and ask there for explanations.

ghost commented 2 years ago

But it works in the detect.py inference file from yolov5 repo. I tested also simplifying the onnx export with onnx-simplifier but that also didn’t help. That’s why I’m sure it’s Opencv related. On yolov5 repo I also saw people having problems loading an exported model (onnx) into Opencv but they just said that they don’t provide any examples to load it in Opencv dnn framework

brmarkus commented 2 years ago

Can you provide more details about your model, please? You are based on a YoloV5s model, but retrained it? Can you describe your code, please? Do you use the example "Object-Detection-using-YOLOv5-and-OpenCV-DNN-in-CPP-and-Python", or another application? Have you done any changes or are you using your own application?

Depending on the used code and which APIs are called the input resolution will be upscaled or downscaled to the NN's expected resolution. If you provide an input picture of 256x256 (and three color-channels) then - depending on which code is used, which APIs are called - it will be upscaled to 640x640 (and three color-channels). It's important to know whether the model expects the color-channel in RGB or BGR order.

Can you maybe share your model and your code and example input files - and then show the output you get and explain, what you think is wrong?

ghost commented 2 years ago

General Setup:

GPU RTX3070ti
Cuda 11.6
Cudnn 8.4.0
OpenCV 4.5.5

So, My model was trained in Google Colab with the Roboflow settings. (https://colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ) I have one Object class to detect (humans). Neural Network size is changed to 256x256 (before training so i didnt just change it afterwards). When i trained my Model for the Darknet format (Yolov4, Yolov4-tiny) I could successfully load it into Opencv and use it (Different way of getting the Detection boxes etc) so the model itself or the lack of labeled data cant really be the issue. My Dataset has 15k labeled images. The Trained Yolov5s model was also tested in detect.py from (https://github.com/ultralytics/yolov5).

CONVERSION: https://github.com/ultralytics/yolov5 export.py. Command i used:

python export.py --weights model.pt --img 256 256 --include onnx
python export.py --weights model.pt --img 256 256 --include onnx --simplify

Both commands successfully converted the PyTorch model to ONNX and both gave me the same Output in VS22

INFERENCE CODE: https://github.com/spmallick/learnopencv/blob/master/Object-Detection-using-YOLOv5-and-OpenCV-DNN-in-CPP-and-Python/yolov5.cpp I used the Example : "Object-Detection-using-YOLOv5-and-OpenCV-DNN-in-CPP-and-Python" and there i used the C++ Code.

const float INPUT_WIDTH = 256;
const float INPUT_HEIGHT = 256;

i loaded my model (yolov5s) and the images i give the neural network are all 256x256 like the neural network input size. blob size is also 256x256. Color format is BGR but i also tried RGB color Format but same issues. After starting the Code from VS22 in Debug mode i get following errors/warnings:

[ INFO:0@0.200] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (726) cv::dnn::dnn4_v20211220::ONNXImporter::populateNet DNN/ONNX: loading ONNX v7 model produced by 'pytorch':1.11.0. Number of nodes = 259, initializers = 213, inputs = 1, outputs = 1
[ INFO:0@0.202] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (642) cv::dnn::dnn4_v20211220::ONNXImporter::parseOperatorSet DNN/ONNX: ONNX opset version = 12
[ INFO:0@0.336] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_208)
[ INFO:0@0.341] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_211)
[ INFO:0@0.342] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_214)
[ INFO:0@0.342] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_217)
[ INFO:0@0.343] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_220)
[ INFO:0@0.348] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_229)
[ INFO:0@0.349] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_232)
[ INFO:0@0.350] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_235)
[ INFO:0@0.350] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_238)
[ INFO:0@0.351] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_241)
[ INFO:0@0.352] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_245)
[ INFO:0@0.352] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_248)
[ INFO:0@0.353] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_252)
[ INFO:0@0.353] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_255)
[ INFO:0@0.355] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_264)
[ INFO:0@0.356] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_267)
[ INFO:0@0.356] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_270)
[ INFO:0@0.357] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_273)
[ INFO:0@0.357] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_276)
[ INFO:0@0.358] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_280)
[ INFO:0@0.359] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_283)
[ INFO:0@0.359] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_287)
[ INFO:0@0.360] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_290)
[ INFO:0@0.361] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_299)
[ INFO:0@0.362] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_302)
[ INFO:0@0.363] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_305)
[ INFO:0@0.368] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_312)
[ INFO:0@0.369] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_315)
[ INFO:0@0.370] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_318)
[ INFO:0@0.371] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_321)
[ INFO:0@0.372] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_329)
[ INFO:0@0.373] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_332)
[ INFO:0@0.375] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_341)
[ INFO:0@0.376] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_344)
[ INFO:0@0.376] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_347)
[ INFO:0@0.378] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_355)
[ INFO:0@0.379] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_358)
[ INFO:0@0.380] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_367)
[ INFO:0@0.381] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_370)
[ INFO:0@0.382] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_373)
[ INFO:0@0.383] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_381)
[ INFO:0@0.384] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_384)
[ INFO:0@0.386] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_388)
[ INFO:0@0.387] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_391)
[ INFO:0@0.387] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_394)
[ INFO:0@0.389] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_402)
[ INFO:0@0.389] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_405)
[ INFO:0@0.390] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_409)
[ INFO:0@0.391] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_412)
[ INFO:0@0.392] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_415)
[ INFO:0@0.393] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(onnx::Mul_423)
[ INFO:0@0.395] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(y)
[ INFO:0@0.401] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(y.3)
[ INFO:0@0.403] global D:\OPENCV\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::
![test](https://user-images.githubusercontent.com/52799640/165961834-25f0b611-f73a-48b6-8b5c-5a516f76d41c.jpg)
dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Sigmoid]:(y.7)

If you could test my model with the given image i'd be happy maybe someone knows how to fix my issues: https://www.mediafire.com/file/i2o09rl2b2qm8cy/best.onnx/file Yolov5s model onnx format https://www.mediafire.com/view/45l4kuya07iqkwo/test.jpg/file TestImage

ct-hub commented 3 months ago

Hello, I also met the same problem, how did I solve it?

brmarkus commented 3 months ago

@ghost and @ct-hub do you want to give it a try with e.g. OpenVINO - either directly or with the OpenCV-DNN-OpenVINO backend?