search

rockchip-linux / rknn-toolkit2

BSD 3-Clause "New" or "Revised" License
828 stars 151 forks source link

Problem in conversion of a STGCN++ .onnx model to .rknn model for RK3588 #315

Open musicgary opened 1 month ago

musicgary commented 1 month ago

Dear all,

I have an .onnx model from STGCN++ model. Using the rknn-toolkit 2 verson 2.0b, when I wanted to convert the model to .rknn for deployment in RK3588, I encountered the following problem: (opset 17)

(rknn2b0) $ python convert_2.py newAction17_cut_9.onnx rk3588 fp newAction17.rknn
I rknn-toolkit2 version: 2.0.0b0+9bab5682
--> Config model
done
--> Loading model
I It is recommended onnx opset 19, but your onnx model opset is 17!
I Model converted from pytorch, 'opset_version' should be set 19 in torch.onnx.export for successful convert!
I Loading : 100%|█████████████████████████████████████████████| 444/444 [00:00<00:00, 265016.50it/s]
W load_onnx: The config.mean_values is None, zeros will be set for input 0!
W load_onnx: The config.std_values is None, ones will be set for input 0!
done
--> Building model
D base_optimize ...
D base_optimize done.
D 
D fold_constant ...
D fold_constant done.
D fold_constant remove nodes = ['/cls_head/Concat_1', '/cls_head/Unsqueeze_6', '/cls_head/Unsqueeze_5', '/cls_head/Unsqueeze_4', '/cls_head/Concat', '/cls_head/Unsqueeze_3', '/cls_head/Unsqueeze_2', '/cls_head/Unsqueeze_1', '/cls_head/Unsqueeze', '/cls_head/Mul', '/cls_head/Gather_4', '/cls_head/Shape_4', '/cls_head/Gather_3', '/cls_head/Shape_3', '/cls_head/Gather_2', '/cls_head/Shape_2', '/cls_head/Gather_1', '/cls_head/Shape_1', '/cls_head/Gather', '/cls_head/Shape', '/backbone/Concat', '/backbone/Unsqueeze_2', '/backbone/Unsqueeze_1', '/backbone/Unsqueeze', '/backbone/Gather_2', '/backbone/Shape_2', '/backbone/Gather_1', '/backbone/Shape_1', '/backbone/Gather', '/backbone/Shape', '/backbone/gcn.9/gcn/Concat', '/backbone/gcn.9/gcn/Unsqueeze_2', '/backbone/gcn.9/gcn/Unsqueeze_1', '/backbone/gcn.9/gcn/Unsqueeze', '/backbone/gcn.9/gcn/Gather_2', '/backbone/gcn.9/gcn/Shape_2', '/backbone/gcn.9/gcn/Gather_1', '/backbone/gcn.9/gcn/Shape_1', '/backbone/gcn.9/gcn/Gather', '/backbone/gcn.9/gcn/Shape', '/backbone/gcn.8/gcn/Concat', '/backbone/gcn.8/gcn/Unsqueeze_2', '/backbone/gcn.8/gcn/Unsqueeze_1', '/backbone/gcn.8/gcn/Unsqueeze', '/backbone/gcn.8/gcn/Gather_2', '/backbone/gcn.8/gcn/Shape_2', '/backbone/gcn.8/gcn/Gather_1', '/backbone/gcn.8/gcn/Shape_1', '/backbone/gcn.8/gcn/Gather', '/backbone/gcn.8/gcn/Shape', '/backbone/gcn.7/gcn/Concat', '/backbone/gcn.7/gcn/Unsqueeze_2', '/backbone/gcn.7/gcn/Unsqueeze_1', '/backbone/gcn.7/gcn/Unsqueeze', '/backbone/gcn.7/gcn/Gather_2', '/backbone/gcn.7/gcn/Shape_2', '/backbone/gcn.7/gcn/Gather_1', '/backbone/gcn.7/gcn/Shape_1', '/backbone/gcn.7/gcn/Gather', '/backbone/gcn.7/gcn/Shape', '/backbone/gcn.6/gcn/Concat', '/backbone/gcn.6/gcn/Unsqueeze_2', '/backbone/gcn.6/gcn/Unsqueeze_1', '/backbone/gcn.6/gcn/Unsqueeze', '/backbone/gcn.6/gcn/Gather_2', '/backbone/gcn.6/gcn/Shape_2', '/backbone/gcn.6/gcn/Gather_1', '/backbone/gcn.6/gcn/Shape_1', '/backbone/gcn.6/gcn/Gather', '/backbone/gcn.6/gcn/Shape', '/backbone/gcn.5/gcn/Concat', '/backbone/gcn.5/gcn/Unsqueeze_2', '/backbone/gcn.5/gcn/Unsqueeze_1', '/backbone/gcn.5/gcn/Unsqueeze', '/backbone/gcn.5/gcn/Gather_2', '/backbone/gcn.5/gcn/Shape_2', '/backbone/gcn.5/gcn/Gather_1', '/backbone/gcn.5/gcn/Shape_1', '/backbone/gcn.5/gcn/Gather', '/backbone/gcn.5/gcn/Shape', '/backbone/gcn.4/gcn/Concat', '/backbone/gcn.4/gcn/Unsqueeze_2', '/backbone/gcn.4/gcn/Unsqueeze_1', '/backbone/gcn.4/gcn/Unsqueeze', '/backbone/gcn.4/gcn/Gather_2', '/backbone/gcn.4/gcn/Shape_2', '/backbone/gcn.4/gcn/Gather_1', '/backbone/gcn.4/gcn/Shape_1', '/backbone/gcn.4/gcn/Gather', '/backbone/gcn.4/gcn/Shape', '/backbone/gcn.3/gcn/Concat', '/backbone/gcn.3/gcn/Unsqueeze_2', '/backbone/gcn.3/gcn/Unsqueeze_1', '/backbone/gcn.3/gcn/Unsqueeze', '/backbone/gcn.3/gcn/Gather_2', '/backbone/gcn.3/gcn/Shape_2', '/backbone/gcn.3/gcn/Gather_1', '/backbone/gcn.3/gcn/Shape_1', '/backbone/gcn.3/gcn/Gather', '/backbone/gcn.3/gcn/Shape', '/backbone/gcn.2/gcn/Concat', '/backbone/gcn.2/gcn/Unsqueeze_2', '/backbone/gcn.2/gcn/Unsqueeze_1', '/backbone/gcn.2/gcn/Unsqueeze', '/backbone/gcn.2/gcn/Gather_2', '/backbone/gcn.2/gcn/Shape_2', '/backbone/gcn.2/gcn/Gather_1', '/backbone/gcn.2/gcn/Shape_1', '/backbone/gcn.2/gcn/Gather', '/backbone/gcn.2/gcn/Shape', '/backbone/gcn.1/gcn/Concat', '/backbone/gcn.1/gcn/Unsqueeze_2', '/backbone/gcn.1/gcn/Unsqueeze_1', '/backbone/gcn.1/gcn/Unsqueeze', '/backbone/gcn.1/gcn/Gather_2', '/backbone/gcn.1/gcn/Shape_2', '/backbone/gcn.1/gcn/Gather_1', '/backbone/gcn.1/gcn/Shape_1', '/backbone/gcn.1/gcn/Gather', '/backbone/gcn.1/gcn/Shape']
D 
D correct_ops ...
D correct_ops done.
D 
D fuse_ops ...
D fuse_ops results:
D     remove_invalid_add: remove node = ['/backbone/gcn.0/Add']
D     bypass_two_reshape: remove node = ['/cls_head/Reshape', '/backbone/Reshape_3']
D     swap_reshape_softmax: remove node = ['/Reshape_1', '/Softmax'], add node = ['/Softmax', '/Reshape_1']
D     unsqueeze_to_4d_bn: remove node = [], add node = ['/backbone/data_bn/BatchNormalization_0_unsqueeze0', '/backbone/data_bn/BatchNormalization_0_unsqueeze1']
D     squeeze_1_in_nd_transpose: remove node = [], add node = ['/backbone/Reshape_1_output_0_squeeze_1_nd_/backbone/Transpose_1', '/backbone/Transpose_1_output_0_squeeze_1_nd_/backbone/Transpose_1']
E build: Catch exception when building RKNN model!
E build: Traceback (most recent call last):
E build:   File "rknn/api/rknn_base.py", line 1993, in rknn.api.rknn_base.RKNNBase.build
E build:   File "rknn/api/graph_optimizer.py", line 1907, in rknn.api.graph_optimizer.GraphOptimizer.fuse_ops
E build:   File "rknn/api/fuse_rules.py", line 14960, in rknn.api.fuse_rules._p_convert_einsum_to_exmatmul
E build: KeyError: 'n'
W If you can't handle this error, please try updating to the latest version of the toolkit2 and runtime from:
  https://console.zbox.filez.com/l/I00fc3 (Pwd: rknn)  Path: RKNPU2_SDK / 2.X.X / develop /
  If the error still exists in the latest version, please collect the corresponding error logs and the model,
  convert script, and input data that can reproduce the problem, and then submit an issue on:
  https://redmine.rock-chips.com (Please consult our sales or FAE for the redmine account)
Build model failed!

I read from the error log and tried to convert using opset 19, still no luck: The error log is as follows:

E load_onnx: Catch exception when loading onnx model: /mnt/nvme0n1p2/home/gary/Documents/demo_8fps_2s_v2.1/newClassifier19.onnx!
E load_onnx: Traceback (most recent call last):
E load_onnx:   File "rknn/api/rknn_base.py", line 1546, in rknn.api.rknn_base.RKNNBase.load_onnx
E load_onnx:   File "rknn/api/rknn_base.py", line 674, in rknn.api.rknn_base.RKNNBase._create_ir_and_inputs_meta
E load_onnx:   File "rknn/api/ir_graph.py", line 70, in rknn.api.ir_graph.IRGraph.__init__
E load_onnx:   File "rknn/api/ir_graph.py", line 555, in rknn.api.ir_graph.IRGraph.rebuild
E load_onnx:   File "/home/gary/anaconda3/envs/rknn2b0/lib/python3.8/site-packages/onnx/checker.py", line 136, in check_model
E load_onnx:     C.check_model(protobuf_string, full_check)
E load_onnx: onnx.onnx_cpp2py_export.checker.ValidationError: Unrecognized attribute: axes for operator ReduceMean
E load_onnx: ==> Context: Bad node spec for node. Name: /cls_head/ReduceMean OpType: ReduceMean
W If you can't handle this error, please try updating to the latest version of the toolkit2 and runtime from:
  https://console.zbox.filez.com/l/I00fc3 (Pwd: rknn)  Path: RKNPU2_SDK / 2.X.X / develop /
  If the error still exists in the latest version, please collect the corresponding error logs and the model,
  convert script, and input data that can reproduce the problem, and then submit an issue on:
  https://redmine.rock-chips.com (Please consult our sales or FAE for the redmine account)
Load model failed!

Does it mean that I have to exclude the einsum operator or ReduceMean operator in deployment in RK3588? Is there any workaround for that? Thanks very much for your help.

happyme531 commented 7 hours ago

Try to use the latest beta version from RKNPU2_SDK, fetch code: rknn