search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.69k stars 616 forks source link

[Bug] Unable to reproduce export to CoreML in lib #2794

Open gustavofuhr opened 1 month ago

gustavofuhr commented 1 month ago

Checklist

Describe the bug

I'm trying to reproduce the tutorial for converting detectors from mmdetection to coreML: https://mmdeploy.readthedocs.io/en/latest/05-supported-backends/coreml.html

I did build lib-torch (I took an hour) but when I'm trying to convert the sample model it still give me an error:

mmdeploy/mmdeploy/mmcv/ops/nms.py", line 566, in multiclass_nms__coreml
    assert ops_available(), 'coreml require custom torchscript ops support.'

Reproduction

python tools/deploy.py \ configs/mmdet/detection/detection_coreml_static-800x1344.py \ /mmdetection_dir/configs/retinanet/retinanet_r18_fpn_1x_coco.py \ /checkpoint/retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth \ /mmdetection_dir/demo/demo.jpg \ --work-dir work_dir/retinanet \ --device cpu \ --dump-info

Environment

I build pytorch 2.3.1, I'm using mmdeploy from main and have a macOS 14.x

Error traceback

07/10 21:56:07 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
07/10 21:56:07 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
07/10 21:56:08 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2torchscript.torch2torchscript in subprocess
07/10 21:56:08 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
07/10 21:56:08 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: ../retinanet_r18_fpn_1x_coco_20220407_171055-614fd399.pth
07/10 21:56:09 - mmengine - INFO - Export PyTorch model to torchscript.
[/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py:257](https://file+.vscode-resource.vscode-cdn.net/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py:257): UserWarning: Detect multiple valid rewriters formmdeploy.mmcv.ops.nms._multiclass_nms, use the first rewriter.
  warnings.warn(
Process Process-2:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/pytorch2torchscript.py", line 63, in torch2torchscript
    trace(
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap
    return self.call_function(func_name_, *args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
    return self.call_function_local(func_name, *args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
    return pipe_caller(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/torch_jit/trace.py", line 107, in trace
    else func(inputs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/.mmdetection/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/.mmdetection/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py", line 85, in single_stage_detector__forward
    return __forward_impl(self, batch_inputs, data_samples=data_samples)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/core/optimizers/function_marker.py", line 266, in g
    rets = f(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py", line 23, in __forward_impl
    output = self.bbox_head.predict(x, data_samples, rescale=False)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 197, in predict
    predictions = self.predict_by_feat(
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/codebase/mmdet/models/dense_heads/base_dense_head.py", line 198, in base_dense_head__predict_by_feat
    return multiclass_nms(
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/core/optimizers/function_marker.py", line 266, in g
    rets = f(*args, **kwargs)
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/mmcv/ops/nms.py", line 517, in multiclass_nms
    return _multiclass_nms(
  File "/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/mmcv/ops/nms.py", line 566, in multiclass_nms__coreml
    assert ops_available(), 'coreml require custom torchscript ops support.'
AssertionError: coreml require custom torchscript ops support.
07/10 21:56:10 - mmengine - ERROR - [/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py](https://file+.vscode-resource.vscode-cdn.net/Users/gustavofuhr/projects/object_detection_ios_comprehensive/third_party/mmdeploy/mmdeploy/apis/core/pipeline_manager.py) - pop_mp_output - 80 - `mmdeploy.apis.pytorch2torchscript.torch2torchscript` with Call id: 0 failed. exit.
gustavofuhr commented 1 month ago

Hello again, I found that a bunch of my problems were due to inconsistencies between the requirements of all the libs involved (coreml, mmdeploy, mmcv, mmdetection, torch, etc.).

To help you guys to improve the documentation and to provide a sample of a model conversion to CoreML (I used rtmdet, since it's one of the best :-)) I did the following ipython notebook: https://github.com/gustavofuhr/ios_object_detection/blob/main/mmdetection_to_coreml/export_mmdetection_coreml.ipynb

I really hope that that helps.