search

open-mmlab / mmocr

OpenMMLab Text Detection, Recognition and Understanding Toolbox
https://mmocr.readthedocs.io/en/dev-1.x/
Apache License 2.0
4.27k stars 743 forks source link

Bad Performance using PANet pre-trained models #465

Closed chunchet-ng closed 3 years ago

chunchet-ng commented 3 years ago

Hi there,

I downloaded the PANet checkpoint for CTW and used them to carry out inference on CTW test set. The results that I got are significantly lower than what's reported on this page. I have been using all default configurations without any changes.

The results that I got are: CTW1500 thr 0.3, recall：0.638, precision: 0.636, hmean:0.637 thr 0.4, recall：0.638, precision: 0.636, hmean:0.637 thr 0.5, recall：0.638, precision: 0.636, hmean:0.637 thr 0.6, recall：0.638, precision: 0.636, hmean:0.637 thr 0.7, recall：0.638, precision: 0.636, hmean:0.637 thr 0.8, recall：0.638, precision: 0.636, hmean:0.637 thr 0.9, recall：0.618, precision: 0.671, hmean:0.643 {'hmean-iou:recall': 0.6182113821138211, 'hmean-iou:precision': 0.6705467372134039, 'hmean-iou:hmean': 0.64331641285956}

I suspect that there is something wrong with the released checkpoints, it seems like it is not trained properly. Based on this CTW training log file, the validation set's H-mean has already reached 0.66035 at 10th epoch, which is higher than what I get through the released checkpoint. Could you please help me to verify this?

gaotongxiao commented 3 years ago

Thanks for the feedback! We have located the buggy PR (https://github.com/open-mmlab/mmocr/pull/448), which had replaced Polygon3 with shapely and affected hmean-iou compuation. We are now fixing this issue. You may reset the codebase to commit 7c1bf45c63e3962bae3ed88ce0fdab967172c07b to avoid this bug. Sorry for causing inconvenience.

chunchet-ng commented 3 years ago

Thanks for the prompt reply, I have confirmed that switching to this commit works. However, I tested PANet -IC15, FCENet -IC15 & CTW, and all of the results look normal to me except PANet -CTW, if the root cause is the library difference, then shouldn't this affects other models/datasets as well?

Also, does mmocr supports the output of results to ICDAR format (in text files)? because I can't seem to find this implementation.

gaotongxiao commented 3 years ago

We've tested the new implementation thoroughly and found that some models, including PANet, would output some "invalid" polygons that are self-touching or self-crossing, tho they make sense from appearances. These polygons are ignored by the new implementation but were not by the old one. We will release a patch to fix this discrepancy soon.

Also, does mmocr supports the output of results to ICDAR format (in text files)? because I can't seem to find this implementation.

Currently not, but any contribution would be appreciated :)

chunchet-ng commented 3 years ago

Okay, I will see what can I do to output them in ICDAR format, thank you once again for your replies, appreciate it very much!

ming-eng commented 2 years ago

panet_r18_fpem_ffm_600e_icdar2015 thr 0.3, recall：0.638, precision: 0.636, hmean:0.637 thr 0.4, recall：0.638, precision: 0.636, hmean:0.637 thr 0.5, recall：0.638, precision: 0.636, hmean:0.637 thr 0.6, recall：0.638, precision: 0.636, hmean:0.637 thr 0.7, recall：0.638, precision: 0.636, hmean:0.637 thr 0.8, recall：0.638, precision: 0.636, hmean:0.637 thr 0.9, recall：0.618, precision: 0.671, hmean:0.643 训练效果太差了 请问你遇到了吗