search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.77k stars 636 forks source link

Core ML: Could not export Python function call 'RoIAlignFunction' #1028

Closed Typiqally closed 2 years ago

Typiqally commented 2 years ago

When trying to export faster_rcnn_regnetx-3.2GF_fpn_1x_coco I get the following:

Command

python mmdeploy/tools/deploy.py \
    mmdeploy/configs/mmdet/detection/detection_coreml_static-800x1344.py \
    mmdetection/configs/regnet/faster_rcnn_regnetx-3.2GF_fpn_1x_coco.py \
    checkpoints/faster_rcnn_regnetx-3.2GF_fpn_1x_coco_20200517_175927-126fd9bf.pth \
    mmdetection/demo/demo.jpg \
    --work-dir work_dir/faster_rcnn_regnetx \
    --device cpu

Result

2022-09-09 12:32:24,161 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2torchscript.torch2torchscript in subprocess
load checkpoint from local path: checkpoints/faster_rcnn_regnetx-3.2GF_fpn_1x_coco_20200517_175927-126fd9bf.pth
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/datasets/utils.py:66: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
  warnings.warn(
2022-09-09 12:32:25,076 - mmdeploy - INFO - Export PyTorch model to torchscript.
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py:255: UserWarning: Detect multiple valid rewriters formmdeploy.codebase.mmdet.core.post_processing._multiclass_nms, use the first rewriter.
  warnings.warn(
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /tmp/pip-req-build-jhk0fkj8/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/models/dense_heads/anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/core/anchor/anchor_generator.py:333: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors`` 
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/core/anchor/anchor_generator.py:369: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` 
  warnings.warn(
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/models/dense_heads/anchor_head.py:116: UserWarning: DeprecationWarning: `num_anchors` is deprecated, for consistency or also use `num_base_priors` instead
  warnings.warn('DeprecationWarning: `num_anchors` is deprecated, '
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py:24: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  img_shape = [int(val) for val in img_shape]
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py:24: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  img_shape = [int(val) for val in img_shape]
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/dense_heads/rpn_head.py:78: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/pytorch/functions/topk.py:28: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  k = torch.tensor(k, device=input.device, dtype=torch.long)
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/core/bbox/delta_xywh_bbox_coder.py:39: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(0) == bboxes.size(0)
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/core/bbox/delta_xywh_bbox_coder.py:41: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(1) == bboxes.size(1)
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:301: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert batch_size == 1, 'batched nms is not supported for now.'
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/mmcv/ops/roi_align.py:83: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert rois.size(1) == 5, 'RoI must be (idx, x1, y1, x2, y2)!'
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/roi_heads/bbox_head.py:113: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  pre_top_k = -1 if post_params.pre_top_k >= bboxes.shape[1] \
2022-09-09 12:32:29,674 - mmdeploy - INFO - Save PyTorch model: work_dir/faster_rcnn_regnetx/end2end.pt.
Process Process-2:
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/pytorch2torchscript.py", line 54, in torch2torchscript
    trace(
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/torch_jit/trace.py", line 106, in trace
    torch.jit.save(ts_model, output_path)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/jit/_serialization.py", line 81, in save
    m.save(f, _extra_files=_extra_files)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/jit/_script.py", line 596, in save
    return self._c.save(str(f), **kwargs)
RuntimeError: 
Could not export Python function call 'RoIAlignFunction'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/mmcv/ops/roi_align.py(215): forward
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/roi_heads/single_level_roi_extractor.py(246): single_roi_extractor__forward
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/optimizers/function_marker.py(261): g
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py(379): wrapper
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/Users/typically/Workspace/vbti-plant-morphology/mmdetection/mmdet/models/roi_heads/standard_roi_head.py(121): _bbox_forward
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/roi_heads/test_mixins.py(51): bbox_test_mixin__simple_test_bboxes
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py(379): wrapper
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/roi_heads/standard_roi_head.py(38): standard_roi_head__simple_test
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py(379): wrapper
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/two_stage.py(59): two_stage_detector__simple_test
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py(379): wrapper
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py(26): __forward_impl
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/optimizers/function_marker.py(261): g
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py(70): base_detector__forward
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py(379): wrapper
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/jit/_trace.py(952): trace_module
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/site-packages/torch/jit/_trace.py(735): trace
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/torch_jit/trace.py(96): trace
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py(107): __call__
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py(275): call_function_local
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py(326): call_function
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py(356): _wrap
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/pytorch2torchscript.py(54): torch2torchscript
/Users/typically/Workspace/vbti-plant-morphology/mmdeploy/mmdeploy/apis/core/pipeline_manager.py(107): __call__
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/process.py(108): run
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/process.py(315): _bootstrap
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/spawn.py(129): _main
/opt/homebrew/Caskroom/miniconda/base/envs/mmlabs/lib/python3.8/multiprocessing/spawn.py(116): spawn_main
<string>(1): <module>

2022-09-09 12:32:29,941 - mmdeploy - ERROR - `mmdeploy.apis.pytorch2torchscript.torch2torchscript` with Call id: 0 failed. exit.

Environment

2022-09-09 12:41:08,773 - mmdeploy - INFO - 

2022-09-09 12:41:08,774 - mmdeploy - INFO - **********Environmental information**********
2022-09-09 12:41:09,061 - mmdeploy - INFO - sys.platform: darwin
2022-09-09 12:41:09,061 - mmdeploy - INFO - Python: 3.8.13 (default, Mar 28 2022, 06:13:39) [Clang 12.0.0 ]
2022-09-09 12:41:09,061 - mmdeploy - INFO - CUDA available: False
2022-09-09 12:41:09,061 - mmdeploy - INFO - GCC: Apple clang version 14.0.0 (clang-1400.0.29.102)
2022-09-09 12:41:09,061 - mmdeploy - INFO - PyTorch: 1.9.0.post2
2022-09-09 12:41:09,061 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 4.2
  - C++ Version: 201402
  - clang 11.1.0
  - OpenMP 201811
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CXX_COMPILER=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1629200524980/_build_env/bin/arm64-apple-darwin20.0.0-clang++, CXX_FLAGS=-ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++  -std=c++14 -fmessage-length=0 -isystem /Users/runner/miniforge3/conda-bld/pytorch-recipe_1629200524980/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/include -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1629200524980/work=/usr/local/src/conda/pytorch-1.9.0 -fdebug-prefix-map=/Users/runner/miniforge3/conda-bld/pytorch-recipe_1629200524980/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac=/usr/local/src/conda-prefix -Wno-deprecated-declarations -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp=libomp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unknown-warning-option -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-aligned-allocation-unavailable -Wno-c++14-extensions -Wno-constexpr-not-const -Wno-missing-braces -Qunused-arguments -fcolor-diagnostics -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-unused-private-field -Wno-missing-braces -Wno-c++14-extensions -Wno-constexpr-not-const, LAPACK_INFO=open, TORCH_VERSION=1.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, 

2022-09-09 12:41:09,061 - mmdeploy - INFO - TorchVision: 0.10.0a0
2022-09-09 12:41:09,061 - mmdeploy - INFO - OpenCV: 4.5.5
2022-09-09 12:41:09,061 - mmdeploy - INFO - MMCV: 1.6.1
2022-09-09 12:41:09,061 - mmdeploy - INFO - MMCV Compiler: clang 14.0.0
2022-09-09 12:41:09,061 - mmdeploy - INFO - MMCV CUDA Compiler: not available
2022-09-09 12:41:09,061 - mmdeploy - INFO - MMDeploy: 0.8.0+b87afb9
2022-09-09 12:41:09,061 - mmdeploy - INFO - 

2022-09-09 12:41:09,061 - mmdeploy - INFO - **********Backend information**********
2022-09-09 12:41:09,268 - mmdeploy - INFO - onnxruntime: None   ops_is_avaliable : False
2022-09-09 12:41:09,269 - mmdeploy - INFO - tensorrt: None  ops_is_avaliable : False
2022-09-09 12:41:09,278 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-09-09 12:41:09,279 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-09-09 12:41:09,280 - mmdeploy - INFO - openvino_is_avaliable: False
2022-09-09 12:41:09,289 - mmdeploy - INFO - snpe_is_available: False
2022-09-09 12:41:09,290 - mmdeploy - INFO - ascend_is_available: False
2022-09-09 12:41:09,666 - mmdeploy - INFO - coreml_is_available: True
2022-09-09 12:41:09,666 - mmdeploy - INFO - 

2022-09-09 12:41:09,666 - mmdeploy - INFO - **********Codebase information**********
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmdet:  2.25.1
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmseg:  None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmcls:  None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmocr:  None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmedit: None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmdet3d:    None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmpose: None
2022-09-09 12:41:09,668 - mmdeploy - INFO - mmrotate:   None

I believe the CoreML backend uses TorchScript to export as an intermediary language to prepare for CoreML conversion. When I use detection_torchscript.py backend directly, this error does not occur. I'm not sure why this is happening with the CoreML backend.

Typiqally commented 2 years ago

Managed to develop a fix for this issue in #1029