Error exporting YoloX model to ONNX

mathmanu commented 2 years ago

I was trying to do onnx export of yolox model - but i got an error.

The script that I used is: python tools/deployment/pytorch2onnx.py \ configs/yolox/yolox_s_8x8_300e_coco.py \ work_dirs/yolox_s_lite/latest.pth \ --output-file output.onnx \ --input-img demo/demo.jpg \ --dynamic-export \ --show \ --verify \ --simplify

But then I get this error: File "/data/ssd/files/a0393608/work/code/github/openmmlab/mmdetection/tools/deployment/pytorch2onnx.py", line 330, in normalize_cfg = parse_normalize_cfg(cfg.test_pipeline) File "/data/ssd/files/a0393608/work/code/github/openmmlab/mmdetection/tools/deployment/pytorch2onnx.py", line 218, in parse_normalize_cfg assert len(norm_config_li) == 1, 'norm_config should only have one' AssertionError: norm_config should only have one

This is because yolox doesn't have input Normalization. I worked around that by providing a dummy normalization. But then I got another error.

File "/data/ssd/files/a0393608/work/code/github/openmmlab/mmdetection/mmdet/models/detectors/base.py", line 169, in forward return self.onnx_export(img[0], img_metas[0]) File "/data/ssd/files/a0393608/work/code/github/openmmlab/mmdetection/mmdet/models/detectors/single_stage.py", line 169, in onnx_export outs, img_metas, with_nms=with_nms) File "/user/a0393608/work/apps/miniconda3/envs/edgeai-mmdetection/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 186, in new_func return old_func(args, **kwargs) File "/data/ssd/files/a0393608/work/code/github/openmmlab/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 492, in onnx_export bboxes = self.bbox_coder.decode( File "/user/a0393608/work/apps/miniconda3/envs/edgeai-mmdetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1178, in getattr type(self).name, name)) AttributeError: 'YOLOXHead' object has no attribute 'bbox_coder'

Environment sys.platform: linux Python: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0] CUDA available: True GPU 0,1,2,3: RTX A4000 CUDA_HOME: /usr/local/cuda NVCC: Build cuda_11.1.TC455_06.29190527_0 GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.10.0+cu111 PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX512
CUDA Runtime 11.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.0.5
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.11.0a0+972e9af OpenCV: 4.5.4 MMCV: 1.4.2 MMCV Compiler: GCC 7.5 MMCV CUDA Compiler: 11.1 MMDetection: 2.20.0+ff9bc39

RunningLeon commented 2 years ago

@mathmanu, Hi, you could try with our new deployment toolkit mmdeploy and yolox is supported for three backends.

mathmanu commented 2 years ago

Hi @RunningLeon, I tried mmdeploy and the onnx export worked. However it would be nice if it works from mmdetection itself - the other single stage models work. It would be good to avoid this additional dependency in our use case.

RunningLeon commented 2 years ago

@mathmanu， Hi, good to know. Deployment feature in mmdetection would be deprecated eventually. Strongly suggest to use mmdeploy.

mathmanu commented 2 years ago

Adding the following to the class YOLOX in mmdetection/mmdet/models/detectors/yolox.py enables the onnx export. The following code was inspired by mmdeploy:

    def onnx_export(self, img, img_metas=None, **kwargs):
        img_metas = img_metas[0] if isinstance(img_metas, (list,tuple)) else img_metas
        assert isinstance(img_metas, dict)
        assert isinstance(img, torch.Tensor)

        is_dynamic_flag = False
        # get origin input shape as tensor to support onnx dynamic shape
        img_shape = torch._shape_as_tensor(img)[2:]
        if not is_dynamic_flag:
            img_shape = [int(val) for val in img_shape]
        img_metas['img_shape'] = img_shape
        img_metas = [img_metas]
        return self.simple_test(img, img_metas, **kwargs)

    def simple_test(self, img, img_metas, **kwargs):
        if torch.onnx.is_in_onnx_export():
            feat = self.extract_feat(img)
            return self.bbox_head.simple_test(feat, img_metas, **kwargs)
        else:
            return super().simple_test(img, img_metas, **kwargs)

It's good to add it since it's a popular model.

Cursky commented 2 years ago

Hello, thank you for sharing your code, but I added code under this class and still can't use torch 2onnx. Can you post your complete code?

Cursky commented 2 years ago

Thank u very much

mathmanu commented 2 years ago

YOLOX does not use image normalization, so another minor change is needed in tools/deployment/pytorch2onnx.py in function parse_normalize_cfg() as shown below.

def parse_normalize_cfg(test_pipeline):
    transforms = None
    for pipeline in test_pipeline:
        if 'transforms' in pipeline:
            transforms = pipeline['transforms']
            break
    assert transforms is not None, 'Failed to find `transforms`'
    norm_config_li = [_ for _ in transforms if _['type'] == 'Normalize']
    assert len(norm_config_li) <= 1, '`norm_config` should only have one'
    norm_config = norm_config_li[0] if len(norm_config_li)>0 else dict(mean=0.0, std=1.0)
    return norm_config

With these two changes, onnx export of YOLOX should work.

Cursky commented 2 years ago

very thank u share u code,I successfully used it to export the onnx model today. I wish you a happy Chinese New Year in advance.

mathmanu commented 2 years ago

@RunningLeon now that it has been verified, could you perfect it and add it to mmdetection? YOLOX is one of the best models in mmdetection interms for accuracy and embedded friendliness - so this will be quite useful.

RunningLeon commented 2 years ago

@RunningLeon now that it has been verified, could you perfect it and add it to mmdetection? YOLOX is one of the best models in mmdetection interms for accuracy and embedded friendliness - so this will be quite useful.

@mathmanu Hi, as mentioned before. You could use mmdeploy.

@mathmanu， Hi, good to know. Deployment feature in mmdetection would be deprecated eventually. Strongly suggest to use mmdeploy.

Cursky commented 2 years ago

@mathmanu Hello, have you tried to use the yolox onnx model exported by your code to predict with OpenCV DNN? Let me refer to it https://github.com/hpc203/yolox-opencv-dnn This code places the exported model, but there is an error reading error: opencv (4.5.4) XXX error (- 215: assertion file) axes getIntvalue(axes.size()-1)<=dims. size in fuction 'Cv::dnn::dnn4_ v20211004:onnximporter::parseUnsqueeze

Cursky commented 2 years ago

@RunningLeon 很感谢你们开发的mmdeploy 但是他在安装时一定需要git 来拉取子模块（好像是这样），但我的工作限制并不允许我这么做，您有什么好的方法帮助我安装mmdeploy吗，非常感谢！！

mathmanu commented 2 years ago

I have checked the accuracy from the exported yolox onnx model using onnxruntime and it was good. But I have not used OpenCV DNN.

Cursky commented 2 years ago

Maybe you can try to deploy it on OpenCV. After all, the hardware is not so advanced. I will try to solve this problem. If I solve it, I will share it with you at the first time

RunningLeon commented 2 years ago

@RunningLeon 很感谢你们开发的mmdeploy 但是他在安装时一定需要git 来拉取子模块（好像是这样），但我的工作限制并不允许我这么做，您有什么好的方法帮助我安装mmdeploy吗，非常感谢！！

@Cursky You can refer to the doc to manually install git submodules of mmdeploy

jiesonshan commented 2 years ago

@mathmanu @Cursky Refer to the above method，I convert onnx with postprocess， but it‘s error！

python deployment/pytorch2onnx.py /data/shanshaojie/models/biandian_v3/yolox_001/yolox_s_8x8_300e_coco.py /data/shanshaojie/models/biandian_v3/yolox_001/best_bbox_mAP_epoch_289.pth --output-file /data/shanshaojie/models/biandian_v3/yolox_001/nr_merge_only_xmbhyc_yolox_s640x640_e001_e289.onnx --verify --shape 640 640

Traceback (most recent call last): File "deployment/pytorch2onnx.py", line 346, in skip_postprocess=args.skip_postprocess) File "deployment/pytorch2onnx.py", line 171, in pytorch2onnx img_list, img_metas=img_meta_list, return_loss=False)[0] File "/home/shanshaojie/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/shanshaojie/project/mmdetection/mmdet/core/export/model_wrappers.py", line 57, in forward if isinstance(scale_factor, (list, tuple, np.ndarray)): IndexError: too many indices for array

dets dimension is error， How can I do it? Thank you

Cursky commented 2 years ago

@jiesonshan Hello, did you modify the code according to mathmanu's method? I can export onnx model after modifying according to his method, but I can't carry out DNN reasoning. You can check it again.

jiesonshan commented 2 years ago

@Cursky , thank you! I modified the code according to mathmanu's method. I convert two onnx models, one with post-processing and the other without post-processing, without post-processing's result is correct. but post-processing's bboxes is error, post-processing's scores is correct.

VJoer commented 2 years ago

I added these two paragraphs as mentioned above, but I still got the same error. @mathmanu @Cursky

python tools/deployment/pytorch2onnx.py configs/yolox/yolox_s_8x8_300e_coco.py work_dirs/yolox_s_8x8_300e_coco/epoch_30.pth --output-file work_dirs/yolox_s_8x8_300e_coco/yolox_s.onnx --input-img demo/demo.jpg --verify --simplify --dynamic-export --shape 640 640

File "/home/jiangxiaowei/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'YOLOXHead' object has no attribute 'bbox_coder'

How can I do it? Thank you

VJoer commented 2 years ago

very thank u share u code,I successfully used it to export the onnx model today. I wish you a happy Chinese New Year in advance.

请问您是用的哪个版本的mmdetection，我在最新版本上按照上面的代码修改了yolox.py和pytorch2onnx.py文件，还是会报错AttributeError: 'YOLOXHead' object has no attribute 'bbox_coder'。麻烦看一下是不是还有其他地方需要修改。谢谢

erfaneshratifar commented 2 years ago

@VJoer i'm having the same issue.

VJoer commented 2 years ago

@VJoer i'm having the same issue. Maybe you could try this

ZwwWayne commented 2 years ago

As MMDeploy already supports YOLOX https://github.com/open-mmlab/mmdeploy. MMDet will no longer maintain the feature of YOLOX ONNX export. Please use MMDeploy and create an issue if you still have questions about that.

chl916185 commented 2 years ago

@mathmanu @Cursky Refer to the above method，I convert onnx with postprocess， but it‘s error！

python deployment/pytorch2onnx.py /data/shanshaojie/models/biandian_v3/yolox_001/yolox_s_8x8_300e_coco.py /data/shanshaojie/models/biandian_v3/yolox_001/best_bbox_mAP_epoch_289.pth --output-file /data/shanshaojie/models/biandian_v3/yolox_001/nr_merge_only_xmbhyc_yolox_s640x640_e001_e289.onnx --verify --shape 640 640

Traceback (most recent call last): File "deployment/pytorch2onnx.py", line 346, in skip_postprocess=args.skip_postprocess) File "deployment/pytorch2onnx.py", line 171, in pytorch2onnx img_list, img_metas=img_meta_list, return_loss=False)[0] File "/home/shanshaojie/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/shanshaojie/project/mmdetection/mmdet/core/export/model_wrappers.py", line 57, in forward if isinstance(scale_factor, (list, tuple, np.ndarray)): IndexError: too many indices for array

dets dimension is error， How can I do it? Thank you

how to do solve it ? @jiesonshan

sitadivon commented 1 year ago

YOLOX does not use image normalization, so another minor change is needed in tools/deployment/pytorch2onnx.py in function parse_normalize_cfg() as shown below.

def parse_normalize_cfg(test_pipeline):
    transforms = None
    for pipeline in test_pipeline:
        if 'transforms' in pipeline:
            transforms = pipeline['transforms']
            break
    assert transforms is not None, 'Failed to find `transforms`'
    norm_config_li = [_ for _ in transforms if _['type'] == 'Normalize']
    assert len(norm_config_li) <= 1, '`norm_config` should only have one'
    norm_config = norm_config_li[0] if len(norm_config_li)>0 else dict(mean=0.0, std=1.0)
    return norm_config

With these two changes, onnx export of YOLOX should work.

I doubt that the dict(mean=0.0, std=1.0) lacks one additional arg to_rgb=False. Without this arg, mmdet will perform cv2.BGR2RGB operation.

open-mmlab / mmdetection

Error exporting YoloX model to ONNX #6999