How/where to get started developing support for models.

[rickvanveen]

I potentially need to convert other models (mmdet, instance segmentation, e.g., yolact, solo) than currently support by mmdeploy. Is there a more detailed guide about how to do this/ how to get started?

[lvhan028]

https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_support_new_models.html It is what we provide right now, no more other detailed guides.

But you can get some points from the code. From mmdeploy.apis.torch2onnx, you can learn the process of how to build a torch model and convert it to onnx. In mmdeploy.codebase.mmdet.deploy.MMDetection, it models the codebase 'mmdetection', which is made up of Task. And a Task is responsible for initializing the pytorch model, creating the model input tensor, inferring, and visualizing the detected objects.

When you try to convert a non-supported detection model by mmdeploy, mmdeploy.codebase.mmdet.models.* is the most important module that you should pay attention to. In this module, you can rewrite the functions of mmdetection that are not suitable for exporting the onnx model. Regarding what the rewriter is and how to code it, you can refer to the link put down at the top

[rickvanveen]

Thank you, this helped! I will try to convert some of these models in the next weeks and report back any problems/findings in this issue?

[rickvanveen]

I am trying to understand how to figure out what needs to be implemented in order to support a new architecture. I have tried to convert an unsupported architecture, i.e., yolact, and it provided the following error:

2022-04-20:11:00:26,root ERROR    [utils.py:43] 'scale_factor'
Traceback (most recent call last):
  File "/root/workspace/mmdeploy/mmdeploy/utils/utils.py", line 38, in target_wrapper
    result = target(*args, **kwargs)
  File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 113, in torch2onnx
    output_file=output_file)
  File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 55, in torch2onnx_impl
    verbose=verbose)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/__init__.py", line 276, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 94, in export
    use_external_data_format=use_external_data_format)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 698, in _export
    dynamic_axes=dynamic_axes)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 456, in _model_to_graph
    use_new_jit_passes)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 417, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 377, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 1139, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 130, in forward
    self._force_outplace,
  File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 116, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/workspace/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper
    return self.func(self, *args, **kwargs)
  File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 69, in base_detector__forward
    return __forward_impl(ctx, self, img, img_metas=img_metas, **kwargs)
  File "/root/workspace/mmdeploy/mmdeploy/core/optimizers/function_marker.py", line 261, in g
    rets = f(*args, **kwargs)
  File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 28, in __forward_impl
    return self.simple_test(img, img_metas, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/models/detectors/yolact.py", line 101, in simple_test
    feat, img_metas, rescale=rescale)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/models/dense_heads/base_dense_head.py", line 360, in simple_test
    return self.simple_test_bboxes(feats, img_metas, rescale=rescale)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/models/dense_heads/dense_test_mixins.py", line 38, in simple_test_bboxes
    *outs, img_metas=img_metas, rescale=rescale)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 186, in new_func
    return old_func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmdet/models/dense_heads/yolact_head.py", line 348, in get_bboxes
    scale_factor = img_metas[img_id]['scale_factor']
KeyError: 'scale_factor'
2022-04-20 11:00:27,141 - mmdeploy - ERROR - torch2onnx failed.

Questions

1. Is this the way to figure out what needs to be supported? Or is there an easier/better approach?
2. What would be the next steps (roughly)?

[lvhan028]

@grimoire Could you please provide more detailed instructions?

[grimoire]

A good startup is trying to add support to part of the model. For example, yolact can be split into backbone, neck, bbox_head. Try to convert it with:

from mmdeploy.core import RewriteContext

# take backbone as example
backbone = yolact_model.backbone
with RewriteContext(cfg=deploy_cfg, backend=backend_name):
      torch.onnx.export(backbone, inputs, dynamic_axes = {...})

Repeat this on neck and bbox_head (and all function/method has been invoked) until export failed. Rewrite all unsupported component until no export failure.

Rewrite a detection head is a challange. Read the implementation in MMDetection and MMDeploy see what is the difference.

[zhangzscn]

Hi @rickvanveen ,Have you successfully converted Yolact to ONNX？

[rickvanveen]

Not yet. Priorities have shifted, so might take a while...