The add layer of DNN does not support broadcasting with different dimensions, causing my model import to fail.

Zhang-Yang-Sustech commented 3 weeks ago

System Information

OpenCV python version: 4.9.0 Operating System / Platform: Windows 11 Python version: 3.11.4

Detailed description

Model: https://drive.google.com/file/d/1kQCpBhQJIeTZAxTOXQBFY87hnOFn_OI4/view?usp=share_link My ONNX model has an add node with inputs as follows: tensor: float32[1,256,64,64] tensor: float32[256,1,1] Error throws by LayerNorm: "CV_CheckEQ(x_ndims - axis, w_ndims, "LayerNorm: shape of weight does not match with given axis and shape of input");" According to the official ONNX documentation, this kind of broadcasting with different dimensions is allowed, but it seems that OpenCV does not support it. onnxdoc

1x256x64x64 op 256x1x1 should be broadcasted correctly to 1x256x64x64. Current OpenCV doesn't support this case.

Steps to reproduce

//later

Issue submission checklist

[X] I report the issue, it's not a question
[X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
[X] I updated to the latest OpenCV version and the issue is still there
[X] There is reproducer code and related data files (videos, images, onnx, etc)

fengyuentau commented 3 weeks ago

Could you share the model for us to reproduce your issue if possible?

WanliZhong commented 3 weeks ago

Update: the model link: https://drive.google.com/file/d/1kQCpBhQJIeTZAxTOXQBFY87hnOFn_OI4/view?usp=share_link

fengyuentau commented 3 weeks ago

Thank you! @WanliZhong

Problem is confirmed. It is due to layer normalization subgraph pattern is matched, so some operators are merged into a single LayerNormalization layer. Although a separate Mul and Add layer supports broadcasting, it is not considered in the implementation of LayerNormalization layer. A recommended suggestion to fix this issue is to store weight and bias as constants in the importer while keep the option of accepting variable weight and bias, then broadcast constant weight and bias when needed in the fianlize stage of layer initiallization.

ONNX Layer Normalization subgraph pattern: https://github.com/opencv/opencv/blob/5b0843728ebd96196dbeef31cfa81c37f04234d5/modules/dnn/src/onnx/onnx_graph_simplifier.cpp#L770-L897

Parser of ONNX Layer Normalization: https://github.com/opencv/opencv/blob/5b0843728ebd96196dbeef31cfa81c37f04234d5/modules/dnn/src/onnx/onnx_importer.cpp#L3157-L3199

Calculate memory shape in the stage of layer initialization: https://github.com/opencv/opencv/blob/5b0843728ebd96196dbeef31cfa81c37f04234d5/modules/dnn/src/layers/layer_norm.cpp#L59-L90

Finalize a layer initialization: https://github.com/opencv/opencv/blob/5b0843728ebd96196dbeef31cfa81c37f04234d5/modules/dnn/src/layers/layer_norm.cpp#L92-L103

Zhang-Yang-Sustech commented 3 weeks ago

This is the simplified original model which will cause the problem. Thank you.

羊习习 @.***

Original Email

Sender:"Yuantao Feng"< @.*** >;

Sent Time:2024/4/23 15:40

To:"opencv/opencv"< @.*** >;

Cc recipient:"Zhang-Yang-Sustech"< @. >;"Author"< @. >;

Subject:Re: [opencv/opencv] The add layer of DNN does not supportbroadcasting with different dimensions, causing my model import to fail.(Issue #25470)

Could you share the model if possible?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

                    从QQ邮箱发来的超大附件    

                                                                    efficientsam_ti_origin_sim.onnx                              (45.6MB, 无限期)                                                                          进入下载页面                          ：https://wx.mail.qq.com/ftn/download?func=3&k=cbca486176c02784f9b91661633239329d30e16361323932111a401500515b515056155502565b1f0c575d024c500e0b5518090705050d00020108550303263251535e08025b5c5c4046590c3e46506d5b475106085c66415d58160e0f5c41266a5a168a31a9fa149da6050383df4b0f56d24bd0&key=cbca486176c02784f9b91661633239329d30e16361323932111a401500515b515056155502565b1f0c575d024c500e0b5518090705050d00020108550303263251535e08025b5c5c4046590c3e46506d5b475106085c66415d58160e0f5c41266a5a168a31a9fa149da6050383df4b0f56d24bd0&code=458aa292&from=

fengyuentau commented 3 weeks ago

Looks like this issue has been resolved in the latest 4.x. Please verify if I am correct. If it has been resolved, @WanliZhong please close this one.

WanliZhong commented 2 weeks ago

Confirmed this bug is fixed in PR https://github.com/opencv/opencv/pull/24808

opencv / opencv