Open willow-hu opened 1 year ago
This issue occurs when we use the flag --save-preds
and MultiDatasetsEvaluator
simultaneously. It appears that these two components are not compatible with each other. You can simply remove --save-preds
when running test code for several datasets.
However, if you want to obtain performance results and save pkl files for each dataset individually, you can use the --save-preds
flag. In this case, avoid using the MultiDatasetsEvaluator
type and instead utilize the Metric
type, as shown below:
val_evaluator = [
dict(type='WordMetric', mode=['exact', 'ignore_case', 'ignore_case_symbol']),
dict(type='CharMetric')
]
Here, I have a useful but somewhat inelegant solution that you may find helpful:
Modify the following lines in the file mmocr/tools/test.py (lines 114 to 125 in version d7c59f3):
Replace it with:
# save predictions
if args.save_preds:
dump_metric = dict(
type='DumpResults',
out_file_path=osp.join(
cfg.work_dir,
f'{osp.basename(args.checkpoint)}_predictions.pkl'))
if isinstance(cfg.test_evaluator, dict) and cfg.test_evaluator['type'] == 'MultiDatasetsEvaluator':
cfg.test_evaluator['metrics'].append(dump_metric)
if isinstance(cfg.test_evaluator['dataset_prefixes'], str):
cfg.test_evaluator['dataset_prefixes'] = [cfg.test_evaluator['dataset_prefixes']]
elif isinstance(cfg.test_evaluator, (list, tuple)):
cfg.test_evaluator = list(cfg.test_evaluator)
cfg.test_evaluator.append(dump_metric)
else:
cfg.test_evaluator = [cfg.test_evaluator, dump_metric]
Prerequisite
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
main branch https://github.com/open-mmlab/mmocr
Environment
The environment is built following the official documentation. (conda, pip)
The message of
collect_env.py
:Reproduces the problem - code sample
The config file I used:
Reproduces the problem - command or script
The script I wrote:
Reproduces the problem - error message
Additional information
I downloaded the datasets via scripts provided by your repo. And I downloaded the model from the link you provided. It seems that the 'registry' in mmocr is not passed to mmengine?