search

open-mmlab / mmocr

OpenMMLab Text Detection, Recognition and Understanding Toolbox
https://mmocr.readthedocs.io/en/dev-1.x/
Apache License 2.0
4.22k stars 739 forks source link

[Bug] val and test #1897

Open SoftSisterRui opened 1 year ago

SoftSisterRui commented 1 year ago

Prerequisite

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmocr

Environment

I followed the official installation commands for the environment configuration.

Reproduces the problem - code sample

My configuration file is as follows. base_ = [ '_base_svtr-tiny.py', '../base/default_runtime.py', '../base/datasets/mjsynth.py', '../base/datasets/synthtext.py', '../base/datasets/cute80.py', '../base/datasets/iiit5k.py', '../base/datasets/svt.py', '../base/datasets/svtp.py', '../base/datasets/icdar2013.py', '../base/datasets/icdar2015.py', '../base/schedules/schedule_adam_base.py', ]

train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=20, val_interval=20)

optim_wrapper = dict( type='OptimWrapper', optimizer=dict( type='AdamW', lr=5 / (10*4) 2048 / 2048, betas=(0.9, 0.99), eps=8e-8, weight_decay=0.05))

param_scheduler = [ dict( type='LinearLR', start_factor=0.5, end_factor=1., end=2, verbose=False, convert_to_iter_based=True), dict( type='CosineAnnealingLR', T_max=19, begin=2, end=20, verbose=False, convert_to_iter_based=True), ]

dataset settings

train_list = [base.mjsynth_sub_textrecog_train] test_list = [ base.mjsynth_textrecog_test ]

val_evaluator = dict( dataset_prefixes=['CUTE80', 'IIIT5K', 'SVT', 'SVTP', 'IC13', 'IC15']) test_evaluator = val_evaluator

train_dataloader = dict( batch_size=512, num_workers=24, persistent_workers=True, pin_memory=True, sampler=dict(type='DefaultSampler', shuffle=True), dataset=dict( type='ConcatDataset', datasets=train_list, pipeline=base.train_pipeline))

val_dataloader = dict( batch_size=128, num_workers=8, persistent_workers=True, pin_memory=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='ConcatDataset', datasets=test_list, pipeline=base.test_pipeline))

test_dataloader = val_dataloader

Reproduces the problem - command or script

python ./tools/test.py ./configs/textrecog/svtr/svtr-tiny_20e_st_mj.py ./work_dirs/svtr-tiny_20e_st_mj/epoch_20.pth python ./tools/test.py ./configs/textrecog/svtr/svtr-tiny_20e_st_mj.py ./work_dirs/svtr-tiny_20e_st_mj/epoch_20.pth --show-dir ./result python ./tools/test.py ./configs/textrecog/svtr/svtr-tiny_20e_st_mj.py ./work_dirs/svtr-tiny_20e_st_mj/epoch_20.pth --save-preds

Reproduces the problem - error message

My training works fine. But after I use the command python . /tools/test.py . /configs/textrecog/svtr/svtr-tiny_20e_st_mj.py . /work_dirs/svtr-tiny_20e_st_mj/epoch_20.pth seems to work fine, but after testing all the images it reports an error. Traceback (most recent call last): File "./tools/test.py", line 141, in main() File "./tools/test.py", line 137, in main runner.test() File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1752, in test metrics = self.test_loop.run() # type: ignore File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 438, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/mmocr/mmocr/evaluation/evaluator/multi_datasets_evaluator.py", line 53, in evaluate assert len(dataset_slices) == len(self.dataset_prefixes) AssertionError

After I use the command python . /tools/test.py . /configs/textrecog/svtr/svtr-tiny_20e_st_mj.py . /work_dirs/svtr-tiny_20e_st_mj/epoch_20.pth --save-preds does not start the test, it reports an error.

Traceback (most recent call last): File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg obj = obj_cls(*args) # type: ignore File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/loops.py", line 413, in init self.evaluator = runner.build_evaluator(evaluator) # type: ignore File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1297, in build_evaluator return Evaluator(evaluator) # type: ignore File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 25, in init self.metrics.append(METRICS.build(metric)) File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/registry.py", line 545, in build return self.build_func(cfg, args, **kwargs, registry=self) File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 100, in build_from_cfg raise KeyError( KeyError: 'MultiDatasetsEvaluator is not in the metric registry. Please check whether the value of MultiDatasetsEvaluator is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./tools/test.py", line 141, in main() File "./tools/test.py", line 137, in main runner.test() File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1745, in test self._test_loop = self.build_test_loop(self._test_loop) # type: ignore File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1548, in build_test_loop loop = LOOPS.build( File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/registry.py", line 545, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 135, in build_from_cfg raise type(e)( KeyError: "class TestLoop in mmengine/runner/loops.py: 'MultiDatasetsEvaluator is not in the metric registry. Please check whether the value of MultiDatasetsEvaluator is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/advanced_tutorials/config.html#import-the-custom-module'"

Additional information

My dataset profile is as follows. mjsynth_textrecog_data_root = 'data/'

mjsynth_textrecog_test = dict( type='OCRDataset', data_root=mjsynth_textrecog_data_root, data_prefix=dict(img_path='test'), ann_file='modified_test.json', test_mode=False, pipeline=None)

mjsynth_sub_textrecog_train = dict( type='OCRDataset', data_root=mjsynth_textrecog_data_root, data_prefix=dict(img_path='train'), ann_file='modified_train.json', test_mode=False, pipeline=None)

Mountchicken commented 1 year ago

Hi @SoftSisterRui This is indeed a bug. You can check this solution