search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.77k stars 636 forks source link

[Bug] DDQDETR export to onnx error #2807

Open vedrusss opened 3 months ago

vedrusss commented 3 months ago

Checklist

Describe the bug

DDQDETR export to ONNX is not working due to unsupported data type - InstanceData The issue has been first time raised here: https://github.com/open-mmlab/mmdeploy/issues/2561 and not resolved yet (but the ticket has been closed due to inactivity).

Reproduction

python tools/torch2onnx.py --work-dir work_dir --device cuda:0 configs/mmdet/detection/detection_onnxruntime_static.py ddq-detr-4scale_swinl_8xb2-30e_coco_joint.py ddq-detr-4sclale_swinin_finetuned_epoch_5.pth test_image.png

Environment

08/08 06:37:17 - mmengine - INFO - 

08/08 06:37:17 - mmengine - INFO - **********Environmental information**********
08/08 06:37:18 - mmengine - INFO - sys.platform: linux
08/08 06:37:18 - mmengine - INFO - Python: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
08/08 06:37:18 - mmengine - INFO - CUDA available: True
08/08 06:37:18 - mmengine - INFO - MUSA available: False
08/08 06:37:18 - mmengine - INFO - numpy_random_seed: 2147483648
08/08 06:37:18 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 2070 Super
08/08 06:37:18 - mmengine - INFO - CUDA_HOME: /usr/local/cuda
08/08 06:37:18 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.6, V11.6.124
08/08 06:37:18 - mmengine - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
08/08 06:37:18 - mmengine - INFO - PyTorch: 1.10.0
08/08 06:37:18 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

08/08 06:37:18 - mmengine - INFO - TorchVision: 0.11.0
08/08 06:37:18 - mmengine - INFO - OpenCV: 4.8.1
08/08 06:37:18 - mmengine - INFO - MMEngine: 0.10.4
08/08 06:37:18 - mmengine - INFO - MMCV: 2.1.0
08/08 06:37:18 - mmengine - INFO - MMCV Compiler: GCC 9.4
08/08 06:37:18 - mmengine - INFO - MMCV CUDA Compiler: 11.6
08/08 06:37:18 - mmengine - INFO - MMDeploy: 1.3.1+
08/08 06:37:18 - mmengine - INFO - 

08/08 06:37:18 - mmengine - INFO - **********Backend information**********
08/08 06:37:18 - mmengine - INFO - tensorrt:    8.2.4.2
08/08 06:37:18 - mmengine - INFO - tensorrt custom ops: Available
08/08 06:37:18 - mmengine - INFO - ONNXRuntime: None
08/08 06:37:18 - mmengine - INFO - ONNXRuntime-gpu: 1.8.1
08/08 06:37:18 - mmengine - INFO - ONNXRuntime custom ops:  Available
08/08 06:37:18 - mmengine - INFO - pplnn:   None
08/08 06:37:18 - mmengine - INFO - ncnn:    None
08/08 06:37:18 - mmengine - INFO - snpe:    None
08/08 06:37:18 - mmengine - INFO - openvino:    None
08/08 06:37:18 - mmengine - INFO - torchscript: 1.10.0
08/08 06:37:18 - mmengine - INFO - torchscript custom ops:  NotAvailable
08/08 06:37:18 - mmengine - INFO - rknn-toolkit:    None
08/08 06:37:18 - mmengine - INFO - rknn-toolkit2:   None
08/08 06:37:18 - mmengine - INFO - ascend:  None
08/08 06:37:18 - mmengine - INFO - coreml:  None
08/08 06:37:18 - mmengine - INFO - tvm: None
08/08 06:37:18 - mmengine - INFO - vacc:    None
08/08 06:37:18 - mmengine - INFO - 

08/08 06:37:18 - mmengine - INFO - **********Codebase information**********
08/08 06:37:18 - mmengine - INFO - mmdet:   3.3.0
08/08 06:37:18 - mmengine - INFO - mmseg:   None
08/08 06:37:18 - mmengine - INFO - mmpretrain:  None
08/08 06:37:18 - mmengine - INFO - mmocr:   None
08/08 06:37:18 - mmengine - INFO - mmagic:  None
08/08 06:37:18 - mmengine - INFO - mmdet3d: None
08/08 06:37:18 - mmengine - INFO - mmpose:  None
08/08 06:37:18 - mmengine - INFO - mmrotate:    None
08/08 06:37:18 - mmengine - INFO - mmaction:    None
08/08 06:37:18 - mmengine - INFO - mmrazor: None
08/08 06:37:18 - mmengine - INFO - mmyolo:  None

Error traceback

08/06 15:40:10 - mmengine - INFO - torch2onnx: 
    model_cfg: /data/tasks/arlo/savant_object/DDQ_convertion/05082024/ddq-detr-4scale_swinl_8xb2-30e_coco_joint.py 
    deploy_cfg: configs/mmdet/detection/detection_onnxruntime_static.py
08/06 15:40:11 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
08/06 15:40:11 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: /data/tasks/arlo/savant_object/DDQ_convertion/05082024/epoch_5.pth
The model and loaded state dict do not match exactly

size mismatch for bbox_head.cls_branches.0.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.0.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.1.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.1.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.2.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.2.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.3.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.3.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.4.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.4.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.5.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.5.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.6.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.6.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.cls_branches.7.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.cls_branches.7.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.0.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.0.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.1.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.1.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.2.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.2.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.3.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.3.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.4.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.4.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.aux_cls_branches.5.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
size mismatch for bbox_head.aux_cls_branches.5.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for dn_query_generator.label_embedding.weight: copying a param with shape torch.Size([17, 256]) from checkpoint, the shape in current model is torch.Size([80, 256]).
unexpected key in source state_dict: decoder.aux_reg_branches.0.0.weight, decoder.aux_reg_branches.0.0.bias, decoder.aux_reg_branches.0.2.weight, decoder.aux_reg_branches.0.2.bias, decoder.aux_reg_branches.0.4.weight, decoder.aux_reg_branches.0.4.bias, decoder.aux_reg_branches.1.0.weight, decoder.aux_reg_branches.1.0.bias, decoder.aux_reg_branches.1.2.weight, decoder.aux_reg_branches.1.2.bias, decoder.aux_reg_branches.1.4.weight, decoder.aux_reg_branches.1.4.bias, decoder.aux_reg_branches.2.0.weight, decoder.aux_reg_branches.2.0.bias, decoder.aux_reg_branches.2.2.weight, decoder.aux_reg_branches.2.2.bias, decoder.aux_reg_branches.2.4.weight, decoder.aux_reg_branches.2.4.bias, decoder.aux_reg_branches.3.0.weight, decoder.aux_reg_branches.3.0.bias, decoder.aux_reg_branches.3.2.weight, decoder.aux_reg_branches.3.2.bias, decoder.aux_reg_branches.3.4.weight, decoder.aux_reg_branches.3.4.bias, decoder.aux_reg_branches.4.0.weight, decoder.aux_reg_branches.4.0.bias, decoder.aux_reg_branches.4.2.weight, decoder.aux_reg_branches.4.2.bias, decoder.aux_reg_branches.4.4.weight, decoder.aux_reg_branches.4.4.bias, decoder.aux_reg_branches.5.0.weight, decoder.aux_reg_branches.5.0.bias, decoder.aux_reg_branches.5.2.weight, decoder.aux_reg_branches.5.2.bias, decoder.aux_reg_branches.5.4.weight, decoder.aux_reg_branches.5.4.bias

08/06 15:40:19 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
08/06 15:40:19 - mmengine - INFO - Export PyTorch model to ONNX: /data/tasks/arlo/savant_object/DDQ_convertion/05082024/work_dir/end2end.onnx.
08/06 15:40:20 - mmengine - WARNING - Can not find torch.nn.functional.scaled_dot_product_attention, function rewrite will not be applied
08/06 15:40:20 - mmengine - WARNING - Can not find torch._C._jit_pass_onnx_autograd_function_process, function rewrite will not be applied
/opt/conda/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/detectors/base_detr.py:85: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  img_shape = [int(val) for val in img_shape]
/opt/conda/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/detectors/base_detr.py:85: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  img_shape = [int(val) for val in img_shape]
/opt/conda/lib/python3.8/site-packages/mmdeploy/core/optimizers/function_marker.py:160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:167: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  output_h = math.ceil(input_h / stride_h)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:168: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  output_w = math.ceil(input_w / stride_w)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:169: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  pad_h = max((output_h - 1) * stride_h +
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:171: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  pad_w = max((output_w - 1) * stride_w +
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:177: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if pad_h > 0 or pad_w > 0:
/opt/conda/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/backbones.py:189: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert L == H * W, 'input feature has wrong size'
/opt/conda/lib/python3.8/site-packages/mmdet/models/backbones/swin.py:282: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  x = x.view(B, H // window_size, window_size, W // window_size,
/opt/conda/lib/python3.8/site-packages/mmdet/models/backbones/swin.py:92: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  C // self.num_heads).permute(2, 0, 3, 1, 4)
/opt/conda/lib/python3.8/site-packages/mmdet/models/backbones/swin.py:267: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  B = int(windows.shape[0] / (H * W / window_size / window_size))
/opt/conda/lib/python3.8/site-packages/mmdet/models/backbones/swin.py:268: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  x = windows.view(B, H // window_size, W // window_size, window_size,
/opt/conda/lib/python3.8/site-packages/mmdet/models/backbones/swin.py:110: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  attn = attn.view(B // nW, nW, self.num_heads, N,
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:414: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert L == H * W, 'input feature has wrong size'
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:427: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  out_h = (H + 2 * self.sampler.padding[0] - self.sampler.dilation[0] *
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:430: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  out_w = (W + 2 * self.sampler.padding[1] - self.sampler.dilation[1] *
/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:2359: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  _verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/positional_encoding.py:103: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  dim_t = self.temperature**(2 * (dim_t // 2) / self.num_feats)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/deformable_detr_layers.py:103: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for lvl, (H, W) in enumerate(spatial_shapes):
/opt/conda/lib/python3.8/site-packages/mmdeploy/pytorch/functions/linspace.py:15: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if steps == 1:
/opt/conda/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /opt/conda/conda-bld/pytorch_1634272068694/work/aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/opt/conda/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:335: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert (spatial_shapes[:, 0] * spatial_shapes[:, 1]).sum() == num_value
/opt/conda/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:351: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if reference_points.shape[-1] == 2:
/opt/conda/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:136: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  value_list = value.split([H_ * W_ for H_, W_ in value_spatial_shapes],
/opt/conda/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:140: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for level, (H_, W_) in enumerate(value_spatial_shapes):
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/deformable_detr.py:489: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for lvl, HW in enumerate(spatial_shapes):
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/deformable_detr.py:490: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  H, W = HW
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/ddq_detr.py:154: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_imgs = len(scores)
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/ddq_detr.py:173: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  torch.ones(len(single_scores), device=single_scores.device),
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:276: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if boxes.size(-1) == 5:
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:293: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  max_coordinate + torch.tensor(1).to(boxes))
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:302: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if boxes_for_nms.shape[0] < split_thr:
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:317: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  for id in torch.unique(idxs):
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:123: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(1) == 4
/opt/conda/lib/python3.8/site-packages/mmcv/ops/nms.py:124: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(0) == scores.size(0)
/opt/conda/lib/python3.8/site-packages/mmdet/models/utils/misc.py:668: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_padding = max_len - len(input_tensor)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/ddq_detr_layers.py:160: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  len(query) * self.cache_dict['num_heads'], 1, 1)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/ddq_detr_layers.py:162: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if reference_points.shape[-1] == 4:
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:71: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  dim_t = temperature**(2 * (dim_t // 2) / num_feats)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:80: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if coord_tensor.size(-1) == 2:
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/utils.py:82: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  elif coord_tensor.size(-1) == 4:
/opt/conda/lib/python3.8/site-packages/mmcv/ops/multi_scale_deform_attn.py:357: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  elif reference_points.shape[-1] == 4:
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/ddq_detr_layers.py:190: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert reference_points.shape[-1] == 4
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/ddq_detr_layers.py:46: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_imgs = len(reference_points)
/opt/conda/lib/python3.8/site-packages/mmdet/models/layers/transformer/ddq_detr_layers.py:67: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  torch.ones(len(ori_index)),
/opt/conda/lib/python3.8/site-packages/mmdeploy/pytorch/functions/tensor_setitem.py:38: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  stop = stop if stop >= 0 else self_shape[i] + stop
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/dino.py:277: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  if len(query) == self.num_queries:
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/dino.py:286: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  hidden_states=inter_states, references=list(references))
/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/dino.py:286: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  hidden_states=inter_states, references=list(references))
/opt/conda/lib/python3.8/site-packages/mmdet/models/dense_heads/ddq_detr_head.py:141: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if reference.shape[-1] == 4:
/opt/conda/lib/python3.8/site-packages/mmdet/models/dense_heads/detr_head.py:604: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  assert len(cls_score) == len(bbox_pred)  # num_queries
/opt/conda/lib/python3.8/site-packages/mmdet/models/dense_heads/detr_head.py:605: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  max_per_img = self.test_cfg.get('max_per_img', len(cls_score))
/opt/conda/lib/python3.8/site-packages/mmdeploy/pytorch/functions/topk.py:28: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  k = torch.tensor(k, device=input.device, dtype=torch.long)
/opt/conda/lib/python3.8/site-packages/mmdet/models/dense_heads/detr_head.py:612: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  bbox_index = indexes // self.num_classes
/opt/conda/lib/python3.8/site-packages/mmengine/structures/instance_data.py:309: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  return len(self.values()[0])
/opt/conda/lib/python3.8/site-packages/mmengine/structures/instance_data.py:155: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  assert len(value) == len(self), 'The length of ' \
Traceback (most recent call last):
  File "tools/torch2onnx.py", line 85, in <module>
    main()
  File "tools/torch2onnx.py", line 47, in main
    torch2onnx(
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/pytorch2onnx.py", line 98, in torch2onnx
    export(
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/onnx/export.py", line 138, in export
    torch.onnx.export(
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/__init__.py", line 316, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 107, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 724, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/opt/conda/lib/python3.8/site-packages/mmdeploy/apis/onnx/optimizer.py", line 27, in model_to_graph__custom_optimizer
    graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 493, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 437, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/opt/conda/lib/python3.8/site-packages/torch/onnx/utils.py", line 388, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py", line 1166, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_trace.py", line 121, in wrapper
    out_vars, _ = _flatten(outs)
RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: InstanceData
bnddwwj commented 1 month ago

尝试使用mmlab/mmyolo/configs/deploy中的py文件代替mmlab/mmdeploy/configs/mmdet/detection的