search

open-mmlab / mmdeploy

OpenMMLab Model Deployment Framework
https://mmdeploy.readthedocs.io/en/latest/
Apache License 2.0
2.77k stars 634 forks source link

Model failure for text recognition [Copy from Wechat Group] #1102

Open vansin opened 2 years ago

vansin commented 2 years ago

大佬们我运行mmdeploy里面demo/python/ocr.py文件加载文本检测的模型成功了，但是文本识别的模型失败了，怎么解决啊？

文本检测用的dbnet文本识别用的satrn

模型已经用deploy.py转换过了

image

e17158d4dec5504f391c8a3c58987ec9_

b7c010a93e60e03db1935002535677cb_

tpoisonooo commented 2 years ago

@lvhan028

vansin commented 2 years ago

环境配置如下

/home/wang/anaconda3/envs/pytorch/bin/python /home/wang/disk/mmlab/mmdeploy/tools/check_env.py
2022-09-26 09:28:00,625 - mmdeploy - INFO -

2022-09-26 09:28:00,625 - mmdeploy - INFO - Environmental information
2022-09-26 09:28:00,821 - mmdeploy - INFO - sys.platform: linux
2022-09-26 09:28:00,821 - mmdeploy - INFO - Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0]
2022-09-26 09:28:00,821 - mmdeploy - INFO - CUDA available: True
2022-09-26 09:28:00,821 - mmdeploy - INFO - GPU 0: NVIDIA GeForce RTX 3070
2022-09-26 09:28:00,821 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2022-09-26 09:28:00,821 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.7, V11.7.99
2022-09-26 09:28:00,821 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-09-26 09:28:00,822 - mmdeploy - INFO - PyTorch: 1.11.0+cu113
2022-09-26 09:28:00,822 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
CuDNN 8.2
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
2022-09-26 09:28:00,822 - mmdeploy - INFO - TorchVision: 0.12.0+cu113
2022-09-26 09:28:00,822 - mmdeploy - INFO - OpenCV: 4.6.0
2022-09-26 09:28:00,822 - mmdeploy - INFO - MMCV: 1.6.1
2022-09-26 09:28:00,822 - mmdeploy - INFO - MMCV Compiler: GCC 9.3
2022-09-26 09:28:00,822 - mmdeploy - INFO - MMCV CUDA Compiler: 11.3
2022-09-26 09:28:00,822 - mmdeploy - INFO - MMDeploy: 0.8.0+fd07b26
2022-09-26 09:28:00,822 - mmdeploy - INFO -

2022-09-26 09:28:00,822 - mmdeploy - INFO - Backend information
2022-09-26 09:28:01,174 - mmdeploy - INFO - onnxruntime: None ops_is_avaliable : False
2022-09-26 09:28:01,209 - mmdeploy - INFO - tensorrt: 8.2.3.0 ops_is_avaliable : True
2022-09-26 09:28:01,220 - mmdeploy - INFO - ncnn: None ops_is_avaliable : False
2022-09-26 09:28:01,221 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-09-26 09:28:01,221 - mmdeploy - INFO - openvino_is_avaliable: False
2022-09-26 09:28:01,232 - mmdeploy - INFO - snpe_is_available: False
2022-09-26 09:28:01,233 - mmdeploy - INFO - ascend_is_available: False
2022-09-26 09:28:01,234 - mmdeploy - INFO - coreml_is_available: False
2022-09-26 09:28:01,234 - mmdeploy - INFO -

2022-09-26 09:28:01,234 - mmdeploy - INFO - Codebase information
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmdet: 2.25.2
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmseg: None
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmcls: None
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmocr: 0.6.1
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmedit: None
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmdet3d: None
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmpose: None
2022-09-26 09:28:01,235 - mmdeploy - INFO - mmrotate: None

Process finished with exit code 0
vansin commented 2 years ago

完整报错信息如下

(pytorch) wang@wang:~/disk/mmlab$ python mmdeploy/demo/python/ocr.py cuda ./data/demo2.jpg --textdet ./mmdeploy_model/dbnet --textrecog ./mmdeploy_model/satrn
[2022-09-26 09:31:13.128] [mmdeploy] [info] [model.cpp:98] Register 'DirectoryModel'
[2022-09-26 09:31:13.200] [mmdeploy] [info] [model.cpp:39] DirectoryModel successfully load model ./mmdeploy_model/dbnet
[2022-09-26 09:31:14.912] [mmdeploy] [warning] [trt_net.cpp:24] TRTNet: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.5.1
[2022-09-26 09:31:15.004] [mmdeploy] [warning] [trt_net.cpp:24] TRTNet: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.5.1
bboxes.shape=(24, 9)
bboxes=[[1.4200000e+02 3.6000000e+02 4.1800000e+02 3.5400000e+02 4.1800000e+02
3.7700000e+02 1.4300000e+02 3.8400000e+02 7.1645886e-01]
[5.7500000e+02 5.3300000e+02 5.7500000e+02 4.9300000e+02 7.2700000e+02
4.9300000e+02 7.2700000e+02 5.3300000e+02 9.1877848e-01]
[2.9900000e+02 5.0700000e+02 5.5300000e+02 5.0200000e+02 5.5400000e+02
5.3800000e+02 2.9900000e+02 5.4400000e+02 9.2758447e-01]
[7.5100000e+02 5.7000000e+02 8.1000000e+02 5.6800000e+02 8.1100000e+02
5.8500000e+02 7.5200000e+02 5.8700000e+02 8.6515439e-01]
[6.1300000e+02 5.7900000e+02 7.5100000e+02 5.7200000e+02 7.5200000e+02
5.8900000e+02 6.1300000e+02 5.9500000e+02 8.5206240e-01]
[5.1900000e+02 5.8300000e+02 5.9300000e+02 5.7700000e+02 5.9500000e+02
5.9700000e+02 5.2100000e+02 6.0400000e+02 8.0449986e-01]
[4.4900000e+02 5.8500000e+02 5.2200000e+02 5.8400000e+02 5.2200000e+02
6.0300000e+02 4.4900000e+02 6.0400000e+02 8.6480856e-01]
[3.0300000e+02 5.9300000e+02 3.4000000e+02 5.9000000e+02 3.4200000e+02
6.1100000e+02 3.0500000e+02 6.1400000e+02 8.4929198e-01]
[3.4200000e+02 5.9200000e+02 4.4900000e+02 5.8900000e+02 4.4900000e+02
6.0500000e+02 3.4200000e+02 6.0900000e+02 8.6253834e-01]
[2.2100000e+02 5.9800000e+02 3.0300000e+02 5.9500000e+02 3.0300000e+02
6.1500000e+02 2.2200000e+02 6.1700000e+02 8.8932532e-01]
[4.8100000e+02 6.6100000e+02 6.6300000e+02 6.4800000e+02 6.6400000e+02
6.6700000e+02 4.8200000e+02 6.8000000e+02 8.6324644e-01]
[6.6900000e+02 6.4900000e+02 7.1500000e+02 6.4600000e+02 7.1600000e+02
6.6600000e+02 6.7000000e+02 6.6900000e+02 8.7314135e-01]
[4.9100000e+02 7.0700000e+02 5.1300000e+02 7.0500000e+02 5.1600000e+02
7.2800000e+02 4.9400000e+02 7.3000000e+02 7.8541392e-01]
[5.1900000e+02 7.0400000e+02 5.8800000e+02 6.9900000e+02 5.9000000e+02
7.2400000e+02 5.2100000e+02 7.2900000e+02 8.6868352e-01]
[4.7100000e+02 7.2800000e+02 4.7100000e+02 7.1000000e+02 4.8900000e+02
7.1000000e+02 4.8900000e+02 7.2800000e+02 7.0788813e-01]
[6.1000000e+02 6.9800000e+02 6.8100000e+02 6.9500000e+02 6.8200000e+02
7.1800000e+02 6.1100000e+02 7.2200000e+02 8.3222836e-01]
[3.3000000e+02 6.6800000e+02 3.9000000e+02 6.6500000e+02 3.9100000e+02
6.8600000e+02 3.3100000e+02 6.8900000e+02 8.6697978e-01]
[2.1800000e+02 6.7600000e+02 3.2400000e+02 6.6900000e+02 3.2500000e+02
6.8900000e+02 2.1900000e+02 6.9600000e+02 8.7046790e-01]
[3.8600000e+02 7.1500000e+02 4.6600000e+02 7.1000000e+02 4.6800000e+02
7.3000000e+02 3.8800000e+02 7.3500000e+02 8.4747285e-01]
[3.4000000e+02 7.2400000e+02 3.5800000e+02 7.1400000e+02 3.6700000e+02
7.3200000e+02 3.4900000e+02 7.4100000e+02 7.3517746e-01]
[2.1600000e+02 7.2200000e+02 2.6800000e+02 7.1900000e+02 2.7000000e+02
7.4400000e+02 2.1700000e+02 7.4800000e+02 8.6029410e-01]
[2.6700000e+02 7.2000000e+02 3.1700000e+02 7.1700000e+02 3.1800000e+02
7.4200000e+02 2.6800000e+02 7.4400000e+02 8.4578389e-01]
[3.6300000e+02 7.3000000e+02 3.7200000e+02 7.1200000e+02 3.9100000e+02
7.2200000e+02 3.8200000e+02 7.4000000e+02 6.2103432e-01]
[6.8200000e+02 6.9500000e+02 8.2400000e+02 6.8600000e+02 8.2600000e+02
7.0900000e+02 6.8300000e+02 7.1800000e+02 8.4725261e-01]]
[2022-09-26 09:31:15.869] [mmdeploy] [info] [model.cpp:39] DirectoryModel successfully load model ./mmdeploy_model/satrn
[2022-09-26 09:31:15.869] [mmdeploy] [warning] [transform.h:45] Cannot find ResizeOCR implementation on platform cuda
[2022-09-26 09:31:15.869] [mmdeploy] [info] [transform.h:49] Fallback ResizeOCR implementation to platform cpu
[2022-09-26 09:31:15.869] [mmdeploy] [error] [compose.cpp:30] Unable to find Transform creator: ToTensorOCR. Available transforms: ["CenterCrop", "Collect", "Compose", "DefaultFormatBundle", "ImageToTensor", "LoadImageFromFile", "Normalize", "Pad", "Resize", "ResizeOCR", "TopDownAffine", "TopDownGetBboxCenterScale"]
[2022-09-26 09:31:15.869] [mmdeploy] [error] [task.cpp:67] error parsing config: {
"context": {
"device": "",
"model": "",
"stream": ""
},
"input": [
"img"
],
"module": "Transform",
"name": "Preprocess",
"output": [
"prep_output"
],
"transforms": [
{
"type": "LoadImageFromFile"
},
{
"height": 32,
"keep_aspect_ratio": false,
"max_width": 100,
"min_width": 100,
"type": "ResizeOCR",
"width_downsample_ratio": 0.25
},
{
"type": "ToTensorOCR"
},
{
"mean": [
0.485,
0.456,
0.406
],
"std": [
0.229,
0.224,
0.225
],
"type": "NormalizeOCR"
},
{
"keys": [
"img"
],
"meta_keys": [
"img_norm_cfg",
"resize_shape",
"filename",
"flip",
"scale_factor",
"valid_ratio",
"flip_direction",
"img_shape",
"ori_filename",
"ori_shape",
"pad_shape"
],
"type": "Collect"
}
],
"type": "Task"
}
[2022-09-26 09:31:15.869] [mmdeploy] [error] [pipeline.cpp:145] Could not create Task: Preprocess
[2022-09-26 09:31:15.869] [mmdeploy] [error] [pipeline.cpp:159] error parsing config: unknown (6) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/graph/pipeline.cpp:146
[2022-09-26 09:31:15.870] [mmdeploy] [error] [handle.h:31] Failed to create pipeline, config: {
"context": {
"device": "",
"stream": ""
},
"pipeline": {
"input": [
"img",
"dets"
],
"output": [
"text_unflat"
],
"tasks": [
{
"input": [
"img",
"dets"
],
"module": "WarpBoxes",
"name": "warp",
"output": [
"patches"
],
"type": "Task"
},
{
"input": [
"patches"
],
"name": "flatten",
"output": [
"patch_flat",
"patch_index"
],
"type": "Flatten"
},
{
"input": [
"patch_flat"
],
"name": "recog",
"output": [
"texts"
],
"params": {
"batch_size": 1,
"model": ""
},
"type": "Inference"
},
{
"input": [
"texts",
"patch_index"
],
"name": "unflatten",
"output": [
"text_unflat"
],
"type": "Unflatten"
}
]
}
}
[2022-09-26 09:31:15.870] [mmdeploy] [error] [pipeline.cpp:27] exception caught: unknown (6) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/apis/c/mmdeploy/handle.h:32
Traceback (most recent call last):
File "mmdeploy/demo/python/ocr.py", line 69, in
main()
File "mmdeploy/demo/python/ocr.py", line 47, in main
recognizer = TextRecognizer(
RuntimeError: failed to create text_recognizer
AllentDan commented 2 years ago

SATRN is not supported by SDK up to now.

irexyc commented 2 years ago

[2022-09-26 09:31:15.869] [mmdeploy] [error] [compose.cpp:30] Unable to find Transform creator: ToTensorOCR. Available transforms: ["CenterCrop", "Collect", "Compose", "DefaultFormatBundle", "ImageToTensor", "LoadImageFromFile", "Normalize", "Pad", "Resize", "ResizeOCR", "TopDownAffine", "TopDownGetBboxCenterScale"]

There is no implementation of ToTensorOCR in sdk. But you can try to replece:

                    {
                        "type": "ToTensorOCR"
                    },

with:

                    {
                        "type": "Normalize",
                        "mean": [
                            0,
                            0,
                            0
                        ],
                        "std": [
                            255.0,
                            255.0,
                            255.0
                        ]
                    },
                    {
                        "type": "ImageToTensor",
                        "keys": [
                            "img"
                        ]
                    },

in pipeline.json of SATRN

wangyang581 commented 2 years ago

我按照您的建议在pipeline.json中加入了上述字段，代码有了新的报错，如下所示 [2022-09-26 10:36:39.729] [mmdeploy] [info] [model.cpp:39] DirectoryModel successfully load model ./mmdeploy_model/satrn [2022-09-26 10:36:39.729] [mmdeploy] [error] [common.h:34] Failed to find creator: ToTensorOCR. Available: ["Flatten", "Inference", "Pipeline", "Task", "Unflatten"] [2022-09-26 10:36:39.729] [mmdeploy] [error] [pipeline.cpp:159] error parsing config: entry not found (8) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/graph/common.h:35 [2022-09-26 10:36:39.729] [mmdeploy] [error] [handle.h:31] Failed to create pipeline, config: { "context": { "device": "", "stream": "" }, "pipeline": { "input": [ "img", "dets" ], "output": [ "text_unflat" ], "tasks": [ { "input": [ "img", "dets" ], "module": "WarpBoxes", "name": "warp", "output": [ "patches" ], "type": "Task" }, { "input": [ "patches" ], "name": "flatten", "output": [ "patch_flat", "patch_index" ], "type": "Flatten" }, { "input": [ "patch_flat" ], "name": "recog", "output": [ "texts" ], "params": { "batch_size": 1, "model": "" }, "type": "Inference" }, { "input": [ "texts", "patch_index" ], "name": "unflatten", "output": [ "text_unflat" ], "type": "Unflatten" } ] } } [2022-09-26 10:36:39.729] [mmdeploy] [error] [pipeline.cpp:27] exception caught: unknown (6) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/apis/c/mmdeploy/handle.h:32 Traceback (most recent call last): File "mmdeploy/demo/python/ocr.py", line 69, in main() File "mmdeploy/demo/python/ocr.py", line 47, in main recognizer = TextRecognizer( RuntimeError: failed to create text_recognizer 我将上述字段添加到pipeline里面task 的字典里面了，是我添加的位置不对吗？

irexyc commented 2 years ago

We can use the ImageToTensor to replace the ToTensorOCR, but currently we only implement CTCConvertor and lack AttnConvertor of postprocess.

kishcs commented 2 years ago

https://github.com/open-mmlab/mmdeploy/issues/1102#issuecomment-1257391990

After adapting this suggested changes, I have another similar error for NormalizeOCR.


[2022-10-04 07:49:03.635] [mmdeploy] [warning] [transform.h:45] Cannot find ResizeOCR implementation on platform cuda
[2022-10-04 07:49:03.635] [mmdeploy] [info] [transform.h:49] Fallback ResizeOCR implementation to platform cpu
[2022-10-04 07:49:03.635] [mmdeploy] [error] [compose.cpp:30] Unable to find Transform creator: NormalizeOCR. Available transforms: ["CenterCrop", "Collect", "Compose", "DefaultFormatBundle", "ImageToTensor", "LoadImageFromFile", "Normalize", "Pad", "Resize", "ResizeOCR", "TopDownAffine", "TopDownGetBboxCenterScale"]
[2022-10-04 07:49:03.635] [mmdeploy] [error] [task.cpp:67] error parsing config: {
  "context": {
    "device": "<any>",

***

  "type": "Task"
}
[2022-10-04 07:49:03.635] [mmdeploy] [error] [pipeline.cpp:145] Could not create Task: Preprocess
[2022-10-04 07:49:03.636] [mmdeploy] [error] [pipeline.cpp:159] error parsing config: unknown (6) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/graph/pipeline.cpp:146
[2022-10-04 07:49:03.636] [mmdeploy] [error] [handle.h:31] Failed to create pipeline, config: {
  "context": {
    "device": "<any>",
    "stream": "<any>"
  },

***

}
[2022-10-04 07:49:03.636] [mmdeploy] [error] [pipeline.cpp:27] exception caught: unknown (6) @ /data1/prebuild/mmdeploy/csrc/mmdeploy/apis/c/mmdeploy/handle.h:32
Traceback (most recent call last):
  File "sdk/example/python/ocr.py", line 69, in <module>
    main()
  File "sdk/example/python/ocr.py", line 56, in main
    recognizer = TextRecognizer(
RuntimeError: failed to create text_recognizer
AllentDan commented 2 years ago

Will close since satrn is not supported by SDK by far.