search

open-mmlab / mmocr

OpenMMLab Text Detection, Recognition and Understanding Toolbox
https://mmocr.readthedocs.io/en/dev-1.x/
Apache License 2.0
4.27k stars 743 forks source link

[Bug] Detection Returns Empty Array #1886

Open LKWaters opened 1 year ago

LKWaters commented 1 year ago

Prerequisite

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmocr

Environment

sys.platform: linux Python: 3.8.16 (default, Jan 17 2023, 23:13:24) [GCC 11.2.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0: Tesla T4 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.7, V11.7.99 GCC: gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 PyTorch: 1.13.1+cu117 PyTorch compiling details: PyTorch built with:

TorchVision: 0.14.1+cu117 OpenCV: 4.5.3 MMEngine: 0.7.2 MMOCR: 1.0.0+d7c59f3

Reproduces the problem - code sample

mmocr_script.py

from mmocr.apis import MMOCRInferencer rec = 'abinet-vision_20e_st-an_mj' det = 'mask-rcnn_resnet50-oclip_fpn_160e_icdar2015' model = MMOCRInferencer(det=self.det, rec=self.rec)

result = model('path/to/local/image.jpg')

Reproduces the problem - command or script

python mmocr_script.py

Reproduces the problem - error message

[2023-04-30 06:39:54,775: ERROR/ForkPoolWorker-1] Task mmocr.ocr.detect[277fe791-f0ba-4e31-8f0c-a6f18759870b] raised unexpected: Exception() Traceback (most recent call last): File "/home/user/mmocr/ocr.py", line 135, in detect result = pipe.read_img(validation, go_words) File "/home/user/mmocr/ocr.py", line 36, in read_img result = model('image.jpg') File "/home/user/mmocr/mmocr/apis/inferencers/mmocr_inferencer.py", line 321, in call preds = self.forward( File "/home/user/mmocr/mmocr/apis/inferencers/mmocr_inferencer.py", line 175, in forward self.textrec_inferencer( File "/home/user/mmocr/mmocr/apis/inferencers/base_mmocr_inferencer.py", line 203, in call for ori_inputs, data in track( File "/home/user/miniconda3/lib/python3.8/site-packages/rich/progress.py", line 168, in track yield from progress.track( File "/home/user/miniconda3/lib/python3.8/site-packages/rich/progress.py", line 1210, in track for value in sequence: File "/home/user/mmocr/mmocr/apis/inferencers/base_mmocr_inferencer.py", line 80, in preprocess yield from map(self.collate_fn, chunked_data) File "/home/user/mmocr/mmocr/apis/inferencers/base_mmocr_inferencer.py", line 99, in _get_chunk_data pipeout = self.pipeline(inputs) File "/home/user/miniconda3/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 59, in call data = t(data) File "/home/user/miniconda3/lib/python3.8/site-packages/mmcv/transforms/base.py", line 12, in call return self.transform(results) File "/home/user/mmocr/mmocr/datasets/transforms/ocr_transforms.py", line 605, in transform results = super().transform(results) File "/home/user/miniconda3/lib/python3.8/site-packages/mmcv/transforms/processing.py", line 257, in transform self._resize_img(results) File "/home/user/mmocr/mmocr/datasets/transforms/ocr_transforms.py", line 554, in _resize_img return super()._resize_img(results) File "/home/user/miniconda3/lib/python3.8/site-packages/mmcv/transforms/processing.py", line 185, in _resize_img img, w_scale, h_scale = mmcv.imresize( File "/home/user/miniconda3/lib/python3.8/site-packages/mmcv/image/geometric.py", line 116, in imresize resized_img = cv2.resize( cv2.error: OpenCV(4.5.3) /tmp/pip-req-build-f51eratu/opencv/modules/imgproc/src/resize.cpp:4051: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

Additional information

The exact image I'm using that fails: Image

The problem occurs because the detection portion of mmocr_inferencer.py is returning polygons that result in an array of shape (0,3,3). So there needs to be a way for detection to not return empty arrays.

I was able to solve this inelegantly by modifying base_mmocr_inferencer.py but hoping someone with more knowledge could find a better solution.

My changes to base_mmocr_inferencer.py From: 98 pipe_out = self.pipeline(inputs_)

To:

98                     if not inputs_.size == 0:
99                         pipe_out = self.pipeline(inputs_)
gaotongxiao commented 1 year ago

Seems like we need to carry out a validity check before passing the polygons to TextRecInferencer. Thanks for reporting

nba2023 commented 9 months ago

I was able to solve this inelegantly by modifying base_mmocr_inferencer.py but hoping someone with more knowledge could find a better solution.

My changes to base_mmocr_inferencer.py From: 98 pipe_out = self.pipeline(inputs_)

To:

98                     if not inputs_.size == 0:
99                         pipe_out = self.pipeline(inputs_)

I changed the original code to your code but got the error: AttributeError: 'str' object has no attribute 'size' Then change it as follows:

98                     if isinstance(inputs_, np.ndarray) and inputs_.size == 0:
99                         pipe_out = self.pipeline(inputs_)
100                        if pipe_out['data_samples'].get('img_path') is None:
101                           pipe_out['data_samples'].set_metainfo(
102                               dict(img_path=f'{self.num_unnamed_imgs}.jpg'))
103                           self.num_unnamed_imgs += 1
104                        chunk_data.append((inputs_, pipe_out))

Still got error:

Traceback (most recent call last):
File "\\wsl$\Ubuntu-20.04\home\test\workspace\mmocr\tools\infer.py", line 100, in <module>
main()
File "\\wsl$\Ubuntu-20.04\home\test\workspace\mmocr\tools\infer.py", line 96, in main
ocr(**call_args)
File "\\wsl$\ubuntu-20.04\home\test\workspace\mmocr\mmocr\apis\inferencers\mmocr_inferencer.py", line 317, in __call__
preds = self.forward(
File "\\wsl$\ubuntu-20.04\home\test\workspace\mmocr\mmocr\apis\inferencers\mmocr_inferencer.py", line 154, in forward
result['det'] = self.textdet_inferencer(
File "\\wsl$\ubuntu-20.04\home\test\workspace\mmocr\mmocr\apis\inferencers\base_mmocr_inferencer.py", line 196, in __call__
for ori_inputs, data in track(
File "C:\Users\test\scoop\apps\python310\current\lib\site-packages\rich\progress.py", line 168, in track
yield from progress.track(
File "C:\Users\test\scoop\apps\python310\current\lib\site-packages\rich\progress.py", line 1210, in track
for value in sequence:
File "\\wsl$\ubuntu-20.04\home\test\workspace\mmocr\mmocr\apis\inferencers\base_mmocr_inferencer.py", line 80, in preprocess
yield from map(self.collate_fn, chunked_data)
File "C:\Users\test\scoop\apps\python310\current\lib\site-packages\mmengine\dataset\utils.py", line 64, in pseudo_collate
data_item = data_batch[0]
IndexError: list index out of range

Any suggestion to fix this bug? Thanks.

LKWaters commented 7 months ago

I tried my old fix and got the same results you did. Another very inelegant solution is to send a single white square when inputs_ is empty.

98                     try:
99                         pipe_out = self.pipeline(inputs_)
100                     except:
101                         pipe_out = self.pipeline(np.zeros([100,100,3],dtype=np.uint8)

This is again just a work around solution, Ill leave the proper solution to someone that knows mmocr better.