Unable to reproduce results on widerface using tinaface

[manisoftwartist]

I am trying to reproduce the widerface results furnished here: https://github.com/Media-Smart/vedadet/tree/main/configs/trainval/tinaface.

I prepared the data as suggested in the Data Preparation section. I also did the filtering step using python configs/trainval/tinaface/filter_widerface_val.py. With the pretrained model of R50-FPN-BN, I ran the evaluation script configs/trainval/tinaface/test_widerface.py. Here are my results:

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3226/3226, 0.5 task/s, elapsed: 5890s, ETA:     0s                                                                                                                                                                  
+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| class | gts   | dets     | recall | ap    |                                                                                                                                                                                                                            
+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| face  | 31957 | 65363592 | 0.995  | 0.916 |                                                                                                                                                                                                                            
+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| mAP   |       |          |        | 0.916 |
+-------+-------+----------+--------+-------+

From my results above, the mAP matches with your published results. But,

1. The number of detections I get is dets=65363592. Why is this difference in dets not affecting the mAP?
2. Why do I get more detections in spite of the filtering step?
3. I do not see the AP for easy, medium and hard subsets. How do I get these results?
4. This evaluation has taken approximately 98 mins to complete on 3226 widerface validation images on GeForce GTX1080. Is there a way I can improve the speed?

[hxcai]

@manisoftwartist

1. It is the characteristic of mAP metric.
2. You mean you have more gt nums after the filtering step?
3. Please refer to this for evaluation by using official tools.
4. You can limit nms_pre, max_per_img, etc to limit output num and speed up.

[manisoftwartist]

@hxcai Thanks for your quick reply. My gts seems to be correct (same as mike's comment in https://github.com/Media-Smart/vedadet/issues/24).

Here is my mAP using tinaface_r50_fpn_bn

+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| class | gts   | dets     | recall | ap    |                                                                                                                                                                                                                            
+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| face  | 31957 | 65363592 | 0.995  | 0.916 |                                                                                                                                                                                                                            
+-------+-------+----------+--------+-------+                                                                                                                                                                                                                            
| mAP   |       |          |        | 0.916 |
+-------+-------+----------+--------+-------+

And when using tinaface_r50_fpn_gn_dcn

+-------+-------+----------+--------+-------+
| class | gts   | dets     | recall | ap    |
+-------+-------+----------+--------+-------+
| face  | 31957 | 10358268 | 0.995  | 0.923 |
+-------+-------+----------+--------+-------+
| mAP   |       |          |        | 0.923 |
+-------+-------+----------+--------+-------+

So, the mAP on the easy, medium and hard subsets are obtained using the official widerface eval tools. So we need matlab to get these numbers!? Right?

[hxcai]

@manisoftwartist Yeah, we run the official matlab code to get result.