search

opencv / opencv

Open Source Computer Vision Library
https://opencv.org
Apache License 2.0
79.2k stars 55.84k forks source link

Cannot load the paddleocr v3 model after converted to onnx #24559

Open stereomatchingkiss opened 1 year ago

stereomatchingkiss commented 1 year ago

System Information

OpenCV version: 4.8.0 operation System : windows11 compiler : visual c++ 2019

Detailed description

Error messages

[ERROR:0@0.021] global onnx_importer.cpp:1064 cv::dnn::dnn4_v20230620::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node!p2o.MatMul.2) from domain='ai.onnx'
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.8.0) Error: Unspecified error (> Node [MatMul@ai.onnx]:(onnx_node!p2o.MatMul.2) parse error: OpenCV(4.8.0) C:\GHA-OCV-1\_work\ci-gha-workflow\ci-gha-workflow\opencv\modules\dnn\src\layers\fully_connected_layer.cpp:156: error: (-2:Unspecified error) in function 'bool __cdecl cv::dnn::FullyConnectedLayerImpl::getMemoryShapes(const class std::vector<class std::vector<int,class std::allocator<int> >,class std::allocator<class std::vector<int,class std::allocator<int> > > > &,const int,class std::vector<class std::vector<int,class std::allocator<int> >,class std::allocator<class std::vector<int,class std::allocator<int> > > > &,class std::vector<class std::vector<int,class std::allocator<int> >,class std::allocator<class std::vector<int,class std::allocator<int> > > > &) const'
> >  (expected: 'inputsTmp[0][i] == inputsTmp[1][i]'), where
> >     'inputsTmp[0][i]' is 1
> > must be equal to
> >     'inputsTmp[1][i]' is 0
> ) in cv::dnn::dnn4_v20230620::ONNXImporter::handleNode, file C:\GHA-OCV-1\_work\ci-gha-workflow\ci-gha-workflow\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 1083

Steps to reproduce

Steps to reproduce:

  1. Install paddle2onnx and downlaod official paddleocrv3
  2. Convert the model by following cmd
paddle2onnx --model_dir ./ch/ch_PP-OCRv3_rec_infer ^
--model_filename inference.pdmodel ^
--params_filename inference.pdiparams ^
--save_file ./ch/rec_onnx/model.onnx ^
--opset_version 11 ^
--input_shape_dict="{'x':[-1,3,-1,-1]}" ^
--enable_onnx_checker True
  1. Run onnxsimplify
import onnx
from onnxsim import simplify

model = onnx.load(input_model_name)

# convert model
model_simp, check = simplify(model)
assert check, "Simplified ONNX model could not be validated"

onnx.save(model_simp, "{}_simple.onnx".format(output_model_name))
  1. load the model by opencv

cv::dnn::readNet("ch_PP-OCRv3_rec_simple.onnx");

Model after onnx simplify Model before onnx simplify

Issue submission checklist

asmorkalov commented 9 months ago

The same issue with both attached models:

./opencv_model_diagnostics -m=./ch_PP-OCRv3_rec_simple.onnx 
[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 5 inputs and 1 outputs: [Slice]:(onnx_node!p2o.Slice.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:1232: error: (-215:Assertion failed) constBlobs.find(node_proto.input(i)) != constBlobs.end() in function 'parseSlice'

[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!p2o.Concat.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/concat_layer.cpp:105: error: (-215:Assertion failed) curShape.size() == outputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Reshape]:(onnx_node!p2o.Reshape.8) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:597: error: (-5:Bad argument) Blob p2o.Concat.1 not found in const blobs in function 'getBlob'

[ERROR:0@0.073] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 1 inputs and 1 outputs: [Transpose]:(onnx_node!p2o.Transpose.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/permute_layer.cpp:162: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.074] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!p2o.Concat.2) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/concat_layer.cpp:109: error: (-201:Incorrect size of input array) Inconsistent shape for ConcatLayer in function 'getMemoryShapes'

[ERROR:0@0.075] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 3 inputs and 1 outputs: [Conv]:(onnx_node!p2o.Conv.34) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/convolution_layer.cpp:396: error: (-2:Unspecified error) Number of input channels should be multiple of 1024 but got 512 in function 'getMemoryShapes'

[ERROR:0@0.082] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 1 inputs and 1 outputs: [Transpose]:(onnx_node!p2o.Transpose.8) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/permute_layer.cpp:162: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.082] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node!p2o.MatMul.24) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/matmul_layer.cpp:65: error: (-2:Unspecified error) in function 'virtual bool cv::dnn::MatMulLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const'
> DNN/MatMul: invalid dimension K (expected: 'K_A == K_B'), where
>     'K_A' is 1
> must be equal to
>     'K_B' is 64
fengyuentau commented 9 months ago

The same issue with both attached models:

./opencv_model_diagnostics -m=./ch_PP-OCRv3_rec_simple.onnx 
[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 5 inputs and 1 outputs: [Slice]:(onnx_node!p2o.Slice.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:1232: error: (-215:Assertion failed) constBlobs.find(node_proto.input(i)) != constBlobs.end() in function 'parseSlice'

[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!p2o.Concat.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/concat_layer.cpp:105: error: (-215:Assertion failed) curShape.size() == outputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.072] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Reshape]:(onnx_node!p2o.Reshape.8) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:597: error: (-5:Bad argument) Blob p2o.Concat.1 not found in const blobs in function 'getBlob'

[ERROR:0@0.073] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 1 inputs and 1 outputs: [Transpose]:(onnx_node!p2o.Transpose.0) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/permute_layer.cpp:162: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.074] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [Concat]:(onnx_node!p2o.Concat.2) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/concat_layer.cpp:109: error: (-201:Incorrect size of input array) Inconsistent shape for ConcatLayer in function 'getMemoryShapes'

[ERROR:0@0.075] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 3 inputs and 1 outputs: [Conv]:(onnx_node!p2o.Conv.34) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/convolution_layer.cpp:396: error: (-2:Unspecified error) Number of input channels should be multiple of 1024 but got 512 in function 'getMemoryShapes'

[ERROR:0@0.082] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 1 inputs and 1 outputs: [Transpose]:(onnx_node!p2o.Transpose.8) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/permute_layer.cpp:162: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

[ERROR:0@0.082] global onnx_importer.cpp:1014 handleNode DNN/ONNX: Potential problem during processing node with 2 inputs and 1 outputs: [MatMul]:(onnx_node!p2o.MatMul.24) from domain='ai.onnx'
OpenCV(4.9.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/matmul_layer.cpp:65: error: (-2:Unspecified error) in function 'virtual bool cv::dnn::MatMulLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const'
> DNN/MatMul: invalid dimension K (expected: 'K_A == K_B'), where
>     'K_A' is 1
> must be equal to
>     'K_B' is 64

I checked both models and found that they have dynamic batch and width. It would not work with dnn if input is of variable shape. Using onnxsim to overwrite input shape can resolve this problem:

$ onnxsim --overwrite-input-shape=1,3,48,48 model.onnx model.fixed.sim.onnx
Simplifying...
Finish! Here is the difference:
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃                    ┃ Original Model ┃ Simplified Model ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ Add                │ 50             │ 50               │
│ AveragePool        │ 1              │ 1                │
│ BatchNormalization │ 32             │ 0                │
│ Clip               │ 27             │ 27               │
│ Concat             │ 2              │ 1                │
│ Constant           │ 356            │ 116              │
│ Conv               │ 36             │ 36               │
│ Div                │ 32             │ 32               │
│ GlobalAveragePool  │ 2              │ 2                │
│ HardSigmoid        │ 2              │ 2                │
│ MatMul             │ 13             │ 13               │
│ Mul                │ 50             │ 43               │
│ Pow                │ 5              │ 5                │
│ ReduceMean         │ 10             │ 10               │
│ Relu               │ 2              │ 2                │
│ Reshape            │ 6              │ 6                │
│ Shape              │ 1              │ 0                │
│ Sigmoid            │ 7              │ 7                │
│ Slice              │ 7              │ 6                │
│ Softmax            │ 3              │ 3                │
│ Sqrt               │ 5              │ 5                │
│ Squeeze            │ 7              │ 7                │
│ Sub                │ 5              │ 5                │
│ Transpose          │ 9              │ 9                │
│ Model Size         │ 10.2MiB        │ 10.1MiB          │
└────────────────────┴────────────────┴──────────────────┘

$ python
Python 3.10.9 (main, Jan 11 2023, 09:18:18) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2 as cv
>>> cv.dnn.readNet("model.fixed.sim.onnx")
< cv2.dnn.Net 0x1283e4450>