search

aim-uofa / AdelaiDet

AdelaiDet is an open source toolbox for multiple instance-level detection and recognition tasks.
https://git.io/AdelaiDet
Other
3.39k stars 653 forks source link

Evaluation Criteria for ABCNet! #116

Closed Lincoln93 closed 4 years ago

Lincoln93 commented 4 years ago

hey, in the abcnet paper, you showed evaluation on test set using F-measure. But I'm confused, how you measure this for evaluating the text recognition. What's the intuition behind. How is it doing recognition evaluation. Can you point some code reference in the abcnet implementation. Is it evaluation both at a once, detection and recognition?

And why not choose TIoU which is another great work from you. It's a detection metrics, isn't it?

Yuliang-Liu commented 4 years ago

@Lincoln93 For end-to-end recognition, we follow the standard ICDAR evaluation protocol. That is:

Correct localisation was assessed in the same way as in detection. Subsequently, the recognition output for correctly localised words was compared to the ground truth transcription and a perfect match was sought.

But in CTW1500, we take all punctuation marks into accout. We strongly recommend you refer to the code, in which we have presented all the details.

As for TIoU, I personally do believe it is much better than the original IoU detection metric. I may consider include it in the evaluation result. Thanks for your reminder.