clovaai / CLEval

CLEval: Character-Level Evaluation for Text Detection and Recognition Tasks
MIT License
184 stars 28 forks source link

Optimize one-to-one match computation #7

Open vincent-leonardo opened 3 years ago

vincent-leonardo commented 3 years ago

Problem: Repeating same computation when calling one-to-one match.
Especially slow when each individual GT/submission file has large number of bounding boxes (>100). The first 2 conditions of the one_to_one_match only need to be computed once.

How to reproduce:

  1. Prepare a gt.zip in which each .txt file contains large number of bounding boxes.
  2. Similarly, the subm.zip can also contains large number of bounding boxes.
  3. Run python script.py -g="gt.zip" -s="subm.zip" [--E2E] [-t 4]

Output (average bbox per gt/subm: 100~):

'Calculated!'
{'Detection': {'hmean': 0.9995194475865179,
               'precision': 0.9992978099060543,
               'recall': 0.999741183604351},
 'Detection_Metadata': {'char_false_pos': 0,
                        'char_miss': 0,
                        'char_overlap': 60,
                        'num_false_pos': 0,
                        'num_merge': 34,
                        'num_split': 35},
 'EndtoEnd': {'hmean': 0.999741183604351,
              'precision': 0.999741183604351,
              'recall': 0.999741183604351,
              'recognition_score': 1.0},
 'EndtoEnd_Metadata': {'char_false_pos': 0.0,
                       'char_miss': 35.0,
                       'num_false_pos': 0,
                       'num_merge': 34,
                       'num_split': 35}}
Evaluated in 226305.144 ms

After optimization:

'Calculated!'
{'Detection': {'hmean': 0.9995194475865179,
               'precision': 0.9992978099060543,
               'recall': 0.999741183604351},
 'Detection_Metadata': {'char_false_pos': 0,
                        'char_miss': 0,
                        'char_overlap': 60,
                        'num_false_pos': 0,
                        'num_merge': 34,
                        'num_split': 35},
 'EndtoEnd': {'hmean': 0.999741183604351,
              'precision': 0.999741183604351,
              'recall': 0.999741183604351,
              'recognition_score': 1.0},
 'EndtoEnd_Metadata': {'char_false_pos': 0.0,
                       'char_miss': 35.0,
                       'num_false_pos': 0,
                       'num_merge': 34,
                       'num_split': 35}}
Evaluated in 11632.926 ms

Explanation & Solutions: the one_to_one_match is called n_gt * n_sumb times.
The first condition relies only on the row (always sum all second axis) while the second condition relies only on the col (always sum the first axis). All of which can just be computed once and reuse for all the n_gt * n_sumb iterations.
We can immediately filter out the indices that doesn't meet the first & second condition and iterate through those that still possible for one-to-one match (and check with the third condition)