search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.61k stars 603 forks source link

Repeated output boxes after converting to ONNX #1585

Closed dungng27 closed 1 year ago

dungng27 commented 1 year ago

Checklist

Describe the bug

My RotatedFCOS model trained using MMRotate outputs similar boxes after converting to onnx using MMDeploy. If i'm not wrong, maybe because when converting, the model fixed the number of ouput boxes to the number of detected boxes in image used for converting and after converting it just repeats one detected box exactly these number of times. Hope someone can help!

Reproduction

The command to convert model using mmdeploy:

python mmdeploy/tools/deploy.py \
  /workspaces/PheNet-BEV/mmdeploy/configs/phenet/rotated-detection_onnxruntime_dynamic.py \
  /workspaces/PheNet-BEV/training_logs/mbnv3l_fr4_conv64_20x/rotated_fcos_kld_mbnv3_fpn_1x_dota_le90.py \
  /workspaces/PheNet-BEV/training_logs/mbnv3l_fr4_conv64_20x/epoch_202.pth \
  /workspaces/PheNet-BEV/data/dataset_parking_phenikaaX/slot_data_256_with_angle/valid/images/90do_img_240.png \
  --work-dir onnx_model/mbnv3l_fr4_conv64_20x

Environment

Some warning when converting:
2022-12-28 10:33:06,817 - mmdeploy - INFO - **********Environmental information**********
2022-12-28 10:33:06,961 - mmdeploy - INFO - sys.platform: linux
2022-12-28 10:33:06,962 - mmdeploy - INFO - Python: 3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0]
2022-12-28 10:33:06,962 - mmdeploy - INFO - CUDA available: True
2022-12-28 10:33:06,962 - mmdeploy - INFO - GPU 0: NVIDIA GeForce GTX 1660 SUPER
2022-12-28 10:33:06,962 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-12-28 10:33:06,962 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.6, V11.6.124
2022-12-28 10:33:06,962 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-12-28 10:33:06,962 - mmdeploy - INFO - PyTorch: 1.10.0
2022-12-28 10:33:06,962 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

2022-12-28 10:33:06,962 - mmdeploy - INFO - TorchVision: 0.11.0
2022-12-28 10:33:06,962 - mmdeploy - INFO - OpenCV: 4.6.0
2022-12-28 10:33:06,962 - mmdeploy - INFO - MMCV: 1.6.2
2022-12-28 10:33:06,962 - mmdeploy - INFO - MMCV Compiler: GCC 9.3
2022-12-28 10:33:06,962 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-12-28 10:33:06,962 - mmdeploy - INFO - MMDeploy: 0.11.0+2702ac7

Error traceback

2022-12-28 10:36:42,939 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
/opt/conda/lib/python3.8/site-packages/mmcv/cnn/bricks/hsigmoid.py:36: UserWarning: In MMCV v1.4.4, we modified the default value of args to align with PyTorch official. Previous Implementation: Hsigmoid(x) = min(max((x + 1) / 2, 0), 1). Current Implementation: Hsigmoid(x) = min(max((x + 3) / 6, 0), 1).
  warnings.warn(
load checkpoint from local path: /workspaces/PheNet-BEV/training_logs/mbnv3l_fr4_conv64_20x/epoch_202.pth
2022-12-28 10:36:44,140 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
2022-12-28 10:36:44,140 - mmdeploy - INFO - Export PyTorch model to ONNX: onnx_model/mbnv3l_fr4_conv64_20x/end2end.onnx.
2022-12-28 10:36:44,196 - mmdeploy - WARNING - Can not find torch._C._jit_pass_onnx_autograd_function_process, function rewrite will not be applied
/root/workspace/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:2359: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
/opt/conda/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /opt/conda/conda-bld/pytorch_1634272068694/work/aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/workspaces/PheNet-BEV/mmrotate/mmrotate/models/dense_heads/rotated_fcos_head.py:594: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
/workspaces/PheNet-BEV/mmrotate/mmrotate/models/dense_heads/rotated_fcos_head.py:603: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if nms_pre > 0 and scores.shape[0] > nms_pre:
/workspaces/PheNet-BEV/mmrotate/mmrotate/core/bbox/coder/distance_angle_point_coder.py:61: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert points.size(0) == pred_bboxes.size(0)
/workspaces/PheNet-BEV/mmrotate/mmrotate/core/bbox/coder/distance_angle_point_coder.py:62: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert points.size(-1) == 2
/workspaces/PheNet-BEV/mmrotate/mmrotate/core/bbox/coder/distance_angle_point_coder.py:63: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(-1) == 5
/workspaces/PheNet-BEV/mmrotate/mmrotate/core/post_processing/bbox_nms_rotated.py:35: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if multi_bboxes.shape[1] > 5:
/workspaces/PheNet-BEV/mmrotate/mmrotate/core/post_processing/bbox_nms_rotated.py:75: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if bboxes.size(-1) == 5:
/root/workspace/mmdeploy/mmdeploy/pytorch/functions/tensor_setitem.py:37: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  stop = stop if stop >= 0 else self_shape[i] + stop
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:448: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if dets.shape[0] == 0:
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
2022-12-28 10:36:53,345 - mmdeploy - INFO - Execute onnx optimize passes.
2022-12-28 10:36:53,565 - mmdeploy - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx

Below is the predicted boxes of an image. As you can see they are overlapped. image

grimoire commented 1 year ago

Rotated FCOS has not been supported yet. https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/03-benchmark/supported_models.md

If you DO want to convert your model, you can add your own rewriter. rotate_anchor_head as example.

dungng27 commented 1 year ago

Thanks for help!