SparseRCNN model convert to ONNX failed

[tengyao-Zhang]

Checklist

• [X] I have searched related issues but cannot get the expected help.
• [X] 2. I have read the FAQ documentation but cannot get the expected help.
• [X] 3. The bug has not been fixed in the latest version.

Describe the bug

I will convert a sparse rcnn model to onnx file, but it failed.

Reproduction

python tools\deploy.py .\configs\mmdet\detection\detection_onnxruntime_dynamic.py ..\mmdetection\configs\sparse_rcnn\sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco.py ..\checkpoints\sparse_rcnn_r50 _fpn_mstrain_480-800_3x_coco_20201218_154234-7bc5c054.pth .\demo\demo.jpg --work-dir .\work_dir --show --dump-info

Environment

2022-10-12 10:10:42,777 - mmdeploy - INFO -

2022-10-12 10:10:42,777 - mmdeploy - INFO - **********Environmental information**********
2022-10-12 10:10:44,877 - mmdeploy - INFO - sys.platform: win32
2022-10-12 10:10:44,877 - mmdeploy - INFO - Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:00) [MSC v.1929 64 bit (AMD64)]
2022-10-12 10:10:44,878 - mmdeploy - INFO - CUDA available: True
2022-10-12 10:10:44,878 - mmdeploy - INFO - GPU 0: NVIDIA GeForce RTX 3080
2022-10-12 10:10:44,878 - mmdeploy - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3
2022-10-12 10:10:44,878 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.3, V11.3.58
2022-10-12 10:10:44,879 - mmdeploy - INFO - MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30146 版
2022-10-12 10:10:44,879 - mmdeploy - INFO - GCC: n/a
2022-10-12 10:10:44,879 - mmdeploy - INFO - PyTorch: 1.12.1
2022-10-12 10:10:44,879 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - C++ Version: 199711
  - MSVC 192829337
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

2022-10-12 10:10:44,881 - mmdeploy - INFO - TorchVision: 0.13.1
2022-10-12 10:10:44,881 - mmdeploy - INFO - OpenCV: 4.6.0
2022-10-12 10:10:44,881 - mmdeploy - INFO - MMCV: 1.6.1
2022-10-12 10:10:44,881 - mmdeploy - INFO - MMCV Compiler: MSVC 192930140
2022-10-12 10:10:44,881 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-10-12 10:10:44,882 - mmdeploy - INFO - MMDeploy: 0.9.0+c35099e
2022-10-12 10:10:44,882 - mmdeploy - INFO -

2022-10-12 10:10:44,882 - mmdeploy - INFO - **********Backend information**********
2022-10-12 10:10:45,243 - mmdeploy - INFO - onnxruntime: 1.8.1  ops_is_avaliable : False
2022-10-12 10:10:45,292 - mmdeploy - INFO - tensorrt: 8.2.3.0   ops_is_avaliable : False
2022-10-12 10:10:45,332 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-10-12 10:10:45,342 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-10-12 10:10:45,352 - mmdeploy - INFO - openvino_is_avaliable: False
2022-10-12 10:10:45,399 - mmdeploy - INFO - snpe_is_available: False
2022-10-12 10:10:45,412 - mmdeploy - INFO - ascend_is_available: False
2022-10-12 10:10:45,421 - mmdeploy - INFO - coreml_is_available: False
2022-10-12 10:10:45,421 - mmdeploy - INFO -

2022-10-12 10:10:45,422 - mmdeploy - INFO - **********Codebase information**********
2022-10-12 10:10:45,426 - mmdeploy - INFO - mmdet:      2.25.2
2022-10-12 10:10:45,426 - mmdeploy - INFO - mmseg:      None
2022-10-12 10:10:45,427 - mmdeploy - INFO - mmcls:      0.23.2
2022-10-12 10:10:45,427 - mmdeploy - INFO - mmocr:      None
2022-10-12 10:10:45,427 - mmdeploy - INFO - mmedit:     None
2022-10-12 10:10:45,427 - mmdeploy - INFO - mmdet3d:    None
2022-10-12 10:10:45,428 - mmdeploy - INFO - mmpose:     None
2022-10-12 10:10:45,428 - mmdeploy - INFO - mmrotate:   None

Error traceback

(mmdeploy) λ python tools\deploy.py .\configs\mmdet\detection\detection_onnxruntime_dynamic.py ..\mmdetection\configs\sparse_rcnn\sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco.py ..\checkpoints\sparse_rcnn_r50 _fpn_mstrain_480-800_3x_coco_20201218_154234-7bc5c054.pth .\demo\XL5047.jpg --work-dir .\work_dir --show --dump-info
d:\mmdetection\mmdetection\mmdet\datasets\utils.py:66: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  warnings.warn(
2022-10-12 10:11:53,035 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
load checkpoint from local path: ..\checkpoints\sparse_rcnn_r50_fpn_mstrain_480-800_3x_coco_20201218_154234-7bc5c054.pth
d:\mmdetection\mmdetection\mmdet\datasets\utils.py:66: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  warnings.warn(
2022-10-12 10:11:54,560 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
2022-10-12 10:11:54,561 - mmdeploy - INFO - Export PyTorch model to ONNX: .\work_dir\end2end.onnx.
d:\mmdetection\mmdeploy\mmdeploy\core\optimizers\function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
d:\mmdetection\mmdetection\mmdet\models\dense_heads\embedding_rpn_head.py:77: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_imgs = len(imgs[0])
d:\mmdetection\mmdetection\mmdet\models\dense_heads\embedding_rpn_head.py:80: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  h, w = meta['img_shape']
d:\mmdetection\mmdetection\mmdet\models\dense_heads\embedding_rpn_head.py:81: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  imgs_whwh.append(imgs[0].new_tensor([[w, h, w, h]]))
d:\mmdetection\mmdetection\mmdet\models\roi_heads\sparse_roi_head.py:317: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if all([proposal.shape[0] == 0 for proposal in proposal_list]):
d:\mmdetection\mmdetection\mmdet\core\bbox\transforms.py:87: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if bboxes.size(0) > 0:
C:\ProgramData\Anaconda3\envs\mmdeploy\lib\site-packages\mmcv\ops\roi_align.py:83: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert rois.size(1) == 5, 'RoI must be (idx, x1, y1, x2, y2)!'
C:\ProgramData\Anaconda3\envs\mmdeploy\lib\site-packages\mmcv\cnn\bricks\wrappers.py:170: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if x.numel() == 0 and obsolete_torch_version(TORCH_VERSION, (1, 5)):
d:\mmdetection\mmdetection\mmdet\models\roi_heads\sparse_roi_head.py:134: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  rois.new_zeros(len(rois)),  # dummy arg
d:\mmdetection\mmdetection\mmdet\models\roi_heads\bbox_heads\bbox_head.py:433: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert img_ids.numel() <= len(img_metas)
d:\mmdetection\mmdetection\mmdet\models\roi_heads\bbox_heads\bbox_head.py:479: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert rois.size(1) == 4 or rois.size(1) == 5, repr(rois.shape)
d:\mmdetection\mmdetection\mmdet\models\roi_heads\bbox_heads\bbox_head.py:485: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert bbox_pred.size(1) == 4
d:\mmdetection\mmdetection\mmdet\models\roi_heads\bbox_heads\bbox_head.py:489: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if rois.size(1) == 4:
d:\mmdetection\mmdeploy\mmdeploy\codebase\mmdet\core\bbox\delta_xywh_bbox_coder.py:39: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(0) == bboxes.size(0)
d:\mmdetection\mmdetection\mmdet\models\roi_heads\bbox_heads\bbox_head.py:453: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  keep_inds[:len(pos_is_gts_)] = pos_keep
d:\mmdetection\mmdeploy\mmdeploy\pytorch\functions\tensor_setitem.py:42: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  elif out.numel() == 1:
d:\mmdetection\mmdeploy\mmdeploy\pytorch\functions\topk.py:28: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  k = torch.tensor(k, device=input.device, dtype=torch.long)
d:\mmdetection\mmdetection\mmdet\models\roi_heads\sparse_roi_head.py:355: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  bbox_pred_per_img = proposal_list[img_id][topk_indices //
d:\mmdetection\mmdetection\mmdet\core\bbox\transforms.py:127: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if bboxes.shape[0] == 0:
d:\mmdetection\mmdetection\mmdet\core\bbox\transforms.py:131: TracerWarning: Converting a tensor to a NumPy array might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  bboxes = bboxes.detach().cpu().numpy()
d:\mmdetection\mmdetection\mmdet\core\bbox\transforms.py:132: TracerWarning: Converting a tensor to a NumPy array might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  labels = labels.detach().cpu().numpy()
2022-10-12 10:12:21,120 - mmdeploy - ERROR - `mmdeploy.apis.pytorch2onnx.torch2onnx` with Call id: 0 failed. exit.

[Qingrenn]

• I think you should rewrite the simple_test function in the SparseRoIHead class (mmdet/models/roi_heads/sparse_roi_head.py) in mmdeploy.
• When you convert the model to ONNX, you should not convert detection results to a list of numpy arrays (mmdetection/mmdetection/mmdet/models/roi_heads/sparse_roi_head.py:365).
• It is a good example that file (mmdeploy/codebase/mmdet/models/roi_heads/standard_roi_head.py) rewrites the simple_test function in the StandardRoIHead class. Just imitate the example to rewirte SparseRoIHead/simple_test.