search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.78k stars 637 forks source link

YOLOX to Onnx failed RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. #2759

Open 99HU opened 6 months ago

99HU commented 6 months ago

Checklist

Describe the bug

i am trying to trans my improved yolox to onnx format,and the yolox is trained by mmdection. I flowed the mmdeploy document-windows to trans my model, but it comes some error which are recored in the Error traceback. i find out that the document of mmdection said only RetinaNet is supported to trans to onnx.So i want to know the problem is due to the unsupport model? if it is how can i trans my model to onnx format? Any Advice can be help,i am looking forward your reply.

Reproduction

from mmdeploy.apis import torch2onnx from mmdeploy.backend.sdk.export_info import export2SDK

img = r'E:\XuyuanFiles\PaperDataSet\Train\images\1133_horzontalflip.jpg' work_dir = r'E:\XuyuanFiles\mmlab\onnx' save_file = 'end2end.onnx' deploy_cfg = 'mmdeploy-main/configs/mmpretrain/classification_onnxruntime_dynamic.py' model_checkpoint=r'E:\XuyuanFiles\mmlab\mmdetection-main\work_dirs\yolox_D\best_coco_bbox_mAP_epoch_400.pth' model_cfg=r'E:\XuyuanFiles\mmlab\mmdetection-main\work_dirs\yolox_D\yolox_D.py' device = 'cpu'

1. convert model to onnx

torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg, model_checkpoint, device)

2. extract pipeline info for sdk use (dump-info)

export2SDK(deploy_cfg, model_cfg, work_dir, pth=model_checkpoint, device=device)

Environment

(PaperLocation) E:\XuyuanFiles\mmlab\mmdeploy-main>python tools/check_env.py
05/07 19:28:45 - mmengine - INFO - 

05/07 19:28:45 - mmengine - INFO - **********Environmental information**********
05/07 19:28:55 - mmengine - INFO - sys.platform: win32
05/07 19:28:55 - mmengine - INFO - Python: 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:01:59) [MSC v.1929 64 bit (AMD64)]
05/07 19:28:55 - mmengine - INFO - CUDA available: True
05/07 19:28:55 - mmengine - INFO - numpy_random_seed: 2147483648
05/07 19:28:55 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
05/07 19:28:55 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6
05/07 19:28:55 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.6, V11.6.55
05/07 19:28:55 - mmengine - INFO - MSVC: 用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.32.31329 版
05/07 19:28:55 - mmengine - INFO - GCC: n/a
05/07 19:28:55 - mmengine - INFO - PyTorch: 1.13.1+cu116
05/07 19:28:55 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - C++ Version: 199711
  - MSVC 192829337
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 11.6
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe
, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/actions-runner/_work/pytorch/pytorch/builder/windows/mkl/include -DNDEBUG -DUSE_KINETO -DL
IBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VE
RSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

05/07 19:28:55 - mmengine - INFO - TorchVision: 0.14.1+cu116
05/07 19:28:55 - mmengine - INFO - OpenCV: 4.8.0
05/07 19:28:55 - mmengine - INFO - MMEngine: 0.9.1
05/07 19:28:55 - mmengine - INFO - MMCV: 2.0.1
05/07 19:28:55 - mmengine - INFO - MMCV Compiler: MSVC 192930148
05/07 19:28:55 - mmengine - INFO - MMCV CUDA Compiler: 11.6
05/07 19:28:55 - mmengine - INFO - MMDeploy: 1.3.1+
05/07 19:28:55 - mmengine - INFO -

05/07 19:28:55 - mmengine - INFO - **********Backend information**********
05/07 19:28:55 - mmengine - INFO - tensorrt:    None
05/07 19:28:55 - mmengine - INFO - ONNXRuntime: 1.8.1
05/07 19:28:55 - mmengine - INFO - ONNXRuntime-gpu:     1.17.1
05/07 19:28:55 - mmengine - INFO - ONNXRuntime custom ops:      Available
05/07 19:28:55 - mmengine - INFO - pplnn:       None
05/07 19:28:55 - mmengine - INFO - ncnn:        None
05/07 19:28:55 - mmengine - INFO - snpe:        None
05/07 19:28:55 - mmengine - INFO - openvino:    None
05/07 19:28:55 - mmengine - INFO - torchscript: 1.13.1+cu116
05/07 19:28:55 - mmengine - INFO - torchscript custom ops:      NotAvailable
05/07 19:28:55 - mmengine - INFO - rknn-toolkit:        None
05/07 19:28:55 - mmengine - INFO - rknn-toolkit2:       None
05/07 19:28:55 - mmengine - INFO - ascend:      None
05/07 19:28:55 - mmengine - INFO - coreml:      None
05/07 19:28:55 - mmengine - INFO - tvm: None
05/07 19:28:55 - mmengine - INFO - vacc:        None
05/07 19:28:55 - mmengine - INFO - 

05/07 19:28:55 - mmengine - INFO - **********Codebase information**********
05/07 19:28:55 - mmengine - INFO - mmdet:       3.2.0
05/07 19:28:55 - mmengine - INFO - mmseg:       None
05/07 19:28:55 - mmengine - INFO - mmpretrain:  1.1.1
05/07 19:28:55 - mmengine - INFO - mmocr:       None
05/07 19:28:55 - mmengine - INFO - mmagic:      None
05/07 19:28:55 - mmengine - INFO - mmdet3d:     None
05/07 19:28:55 - mmengine - INFO - mmpose:      1.2.0
05/07 19:28:55 - mmengine - INFO - mmrotate:    None
05/07 19:28:55 - mmengine - INFO - mmaction:    None
05/07 19:28:55 - mmengine - INFO - mmrazor:     None
05/07 19:28:55 - mmengine - INFO - mmyolo:      0.6.0

Error traceback

C:\Users\27972\.conda\envs\PaperLocation\python.exe E:/XuyuanFiles/mmlab/trans.py
05/07 19:14:30 - mmengine - WARNING - Failed to search registry with scope "mmpretrain" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmpretrain" is a correct scope, or whether the registry is initialized.
05/07 19:14:30 - mmengine - WARNING - Failed to search registry with scope "mmpretrain" in the "mmpretrain_tasks" registry tree. As a workaround, the current "mmpretrain_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmpretrain" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: E:\XuyuanFiles\mmlab\mmdetection-main\work_dirs\yolox_D\best_coco_bbox_mAP_epoch_400.pth
05/07 19:14:30 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
05/07 19:14:30 - mmengine - INFO - Export PyTorch model to ONNX: E:\XuyuanFiles\mmlab\onnx\end2end.onnx.
05/07 19:14:30 - mmengine - WARNING - Can not find torch.nn.functional.scaled_dot_product_attention, function rewrite will not be applied
stem torch.Size([1, 32, 128, 128]
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\TensorShape.cpp:3191.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmengine\structures\instance_data.py:306: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  return len(self.values()[0])
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmengine\structures\instance_data.py:152: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  assert len(value) == len(self), 'The length of ' \
E:\XuyuanFiles\mmlab\mmdetection-main\mmdet\models\dense_heads\yolox_head.py:397: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if with_nms and results.bboxes.numel() > 0:
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmcv\ops\nms.py:276: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if boxes.size(-1) == 5:
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmcv\ops\nms.py:293: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  max_coordinate + torch.tensor(1).to(boxes))
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmcv\ops\nms.py:302: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if boxes_for_nms.shape[0] < split_thr:
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmcv\ops\nms.py:123: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(1) == 4
C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmcv\ops\nms.py:124: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert boxes.size(0) == scores.size(0)
Traceback (most recent call last):
  File "E:/XuyuanFiles/mmlab/trans.py", line 15, in <module>
    torch2onnx(img, work_dir, save_file, deploy_cfg, model_cfg,
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\pytorch2onnx.py", line 98, in torch2onnx
    export(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\core\pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\onnx\export.py", line 138, in export
    torch.onnx.export(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\onnx\utils.py", line 504, in export
    _export(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\onnx\utils.py", line 1529, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\mmdeploy\apis\onnx\optimizer.py", line 27, in model_to_graph__custom_optimizer
    graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\onnx\utils.py", line 1111, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\onnx\utils.py", line 987, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\onnx\utils.py", line 891, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\jit\_trace.py", line 1184, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\jit\_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "C:\Users\27972\.conda\envs\PaperLocation\lib\site-packages\torch\jit\_trace.py", line 121, in wrapper
    out_vars, _ = _flatten(outs)
RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: DetDataSample

Process finished with exit code 1
Howard9112 commented 4 months ago

has the same issue

FishSmallL commented 2 months ago

how to do, please help

bnddwwj commented 1 month ago

how to do, please help

Excuse me, has the problem been solved