search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.61k stars 603 forks source link

[Bug] Cannot run ONNX converted MMOCR model (SAR) #2660

Open decadance-dance opened 5 months ago

decadance-dance commented 5 months ago

Checklist

Describe the bug

I converted the SAR model to the ONNX format. I'm running a simple inference using Python SDK and getting an error: Segmentation fault.

Reproduction

mim download mmocr --config sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real --dest .
python tools/deploy.py     configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py     sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real.py     sar_resnet31_parallel-deco
der_5e_st-sub_mj-sub_sa_real_20220915_171910-04eb4e75.pth     demo/resources/text_recog.jpg     --work-dir mmdeploy
_models/mmocr/sar/ort     --device cpu     --show     --dump-info

python:

import cv2

# ================= init model ====================
text_recogizer = TextRecognizer(path='mmdeploy/mmdeploy_models/mmocr/sar/ort')

# ================= load image source ====================
# Load an image with opencv
img = cv2.imread('data/test2.jpg')

# ================= usage ====================
# With an image which represents only one word. 
output = text_recogizer(img)

data/test2.jpg: test2

Environment

01/25 00:08:11 - mmengine - INFO - 

01/25 00:08:11 - mmengine - INFO - **********Environmental information**********
01/25 00:08:11 - mmengine - INFO - sys.platform: linux
01/25 00:08:11 - mmengine - INFO - Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
01/25 00:08:11 - mmengine - INFO - CUDA available: False
01/25 00:08:11 - mmengine - INFO - MUSA available: False
01/25 00:08:11 - mmengine - INFO - numpy_random_seed: 2147483648
01/25 00:08:11 - mmengine - INFO - GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
01/25 00:08:11 - mmengine - INFO - PyTorch: 1.13.0+cpu
01/25 00:08:11 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

01/25 00:08:11 - mmengine - INFO - TorchVision: 0.14.0+cpu
01/25 00:08:11 - mmengine - INFO - OpenCV: 4.9.0
01/25 00:08:11 - mmengine - INFO - MMEngine: 0.10.3
01/25 00:08:11 - mmengine - INFO - MMCV: 2.0.0rc4
01/25 00:08:11 - mmengine - INFO - MMCV Compiler: GCC 7.3
01/25 00:08:11 - mmengine - INFO - MMCV CUDA Compiler: not available
01/25 00:08:11 - mmengine - INFO - MMDeploy: 1.3.1+bc75c9d
01/25 00:08:11 - mmengine - INFO - 

01/25 00:08:11 - mmengine - INFO - **********Backend information**********
01/25 00:08:11 - mmengine - INFO - tensorrt:    None
01/25 00:08:11 - mmengine - INFO - ONNXRuntime: 1.16.3
01/25 00:08:11 - mmengine - INFO - ONNXRuntime-gpu:     None
01/25 00:08:11 - mmengine - INFO - ONNXRuntime custom ops:      Available
01/25 00:08:11 - mmengine - INFO - pplnn:       None
01/25 00:08:11 - mmengine - INFO - ncnn:        None
01/25 00:08:11 - mmengine - INFO - snpe:        None
01/25 00:08:12 - mmengine - INFO - openvino:    None
01/25 00:08:12 - mmengine - INFO - torchscript: 1.13.0+cpu
01/25 00:08:12 - mmengine - INFO - torchscript custom ops:      NotAvailable
01/25 00:08:12 - mmengine - INFO - rknn-toolkit:        None
01/25 00:08:12 - mmengine - INFO - rknn-toolkit2:       None
01/25 00:08:12 - mmengine - INFO - ascend:      None
01/25 00:08:12 - mmengine - INFO - coreml:      None
01/25 00:08:12 - mmengine - INFO - tvm: None
01/25 00:08:12 - mmengine - INFO - vacc:        None
01/25 00:08:12 - mmengine - INFO - 

01/25 00:08:12 - mmengine - INFO - **********Codebase information**********
01/25 00:08:12 - mmengine - INFO - mmdet:       3.1.0
01/25 00:08:12 - mmengine - INFO - mmseg:       None
01/25 00:08:12 - mmengine - INFO - mmpretrain:  None
01/25 00:08:12 - mmengine - INFO - mmocr:       1.0.1
01/25 00:08:12 - mmengine - INFO - mmagic:      None
01/25 00:08:12 - mmengine - INFO - mmdet3d:     None
01/25 00:08:12 - mmengine - INFO - mmpose:      None
01/25 00:08:12 - mmengine - INFO - mmrotate:    None
01/25 00:08:12 - mmengine - INFO - mmaction:    None
01/25 00:08:12 - mmengine - INFO - mmrazor:     None
01/25 00:08:12 - mmengine - INFO - mmyolo:      None

Error traceback

loading libmmdeploy_ort_net.so ...
[2024-01-25 00:03:28.096] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "/home/kllis/mmdeploy/mmdeploy_models/mmocr/sar/ort"
[2024-01-25 00:03:28.097] [mmdeploy] [error] [compose.cpp:37] Unable to find Transform creator: PadToWidth. Available transforms: [("CenterCrop", 0), ("Collect", 0), ("Compose", 0), ("DefaultFormatBundle", 0), ("FormatShape", 0), ("ImageToTensor", 0), ("LetterResize", 0), ("Lift", 0), ("LoadImageFromFile", 0), ("Normalize", 0), ("Pad", 0), ("RescaleToHeight", 0), ("Resize", 0), ("ResizeOCR", 0), ("ShortScaleAspectJitter", 0), ("TenCrop", 0), ("ThreeCrop", 0), ("TopDownAffine", 0), ("TopDownGetBboxCenterScale", 0)]
[2024-01-25 00:03:28.097] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "img"
  ],
  "module": "Transform",
  "name": "Preprocess",
  "output": [
    "prep_output"
  ],
  "transforms": [
    {
      "type": "LoadImageFromFile"
    },
    {
      "height": 48,
      "max_width": 160,
      "min_width": 48,
      "type": "RescaleToHeight",
      "width_divisor": 4
    },
    {
      "type": "PadToWidth",
      "width": 160
    },
    {
      "mean": [
        127,
        127,
        127
      ],
      "std": [
        127,
        127,
        127
      ],
      "to_rgb": false,
      "type": "Normalize"
    },
    {
      "size_divisor": 1,
      "type": "Pad"
    },
    {
      "type": "DefaultFormatBundle"
    },
    {
      "keys": [
        "img"
      ],
      "meta_keys": [
        "pad_shape",
        "img_shape",
        "img_norm_cfg",
        "flip_direction",
        "ori_filename",
        "ori_shape",
        "valid_ratio",
        "scale_factor",
        "filename",
        "flip",
        "img_path"
      ],
      "type": "Collect"
    }
  ],
  "type": "Task"
}
[2024-01-25 00:03:29.027] [mmdeploy] [info] [inference.cpp:54] ["img"] <- ["patches"]
[2024-01-25 00:03:29.027] [mmdeploy] [info] [inference.cpp:65] ["post_output"] -> ["texts"]
Segmentation fault
tuclen3 commented 4 months ago

Hi @decadance-dance, I also have the same problem with master model. Have you fixed this bug yet? Can you give me a way to fix it? Thanks

ygl89 commented 1 month ago

I have the same issue. The reason of this issue is that PadToWidth is the specific transform process in mmocr but mmdeploy do not support it. So we can use resize before this issue solved.