Question about score threshold of Bbox Branch

[jylins]

Q1: When I set the score threshold to 0.05 as maskrcnn default, the precision was very low. Then I set the score threshold to 0.5, the F-measure matches the proposed score(88.20% on ICDAR 2015 test set), but the recall and the precision do not match the score on paper.	Method	Precision	Recall	F-Measure
Baseline of PMTD	85.84	90.55	88.14
Our Baseline	92.50	84.20	88.20

Q2: Have you do the ablation study on Data Augmentation, RPN Anchor and OHEM. In my experiments, Data Augmentation and OHEM improve the performance, but modification for RPN Anchor does not work.

[JingChaoLiu]

Q1: When I set the score threshold to 0.05 as maskrcnn default, the precision was very low. Then I set the score threshold to 0.5, the F-measure matches the proposed score(88.20% on ICDAR 2015 test set), but the recall and the precision do not match the score on paper.	Method	Precision	Recall	F-Measure
Baseline of PMTD	85.84	90.55	88.14
Our Baseline	92.50	84.20	88.20

A1: Except for the NMS for bounding box, we also do NMS for the quadrilaterals outputted by plane clustering. Consequently, in the test stage, the full pipeline is:

1. image -> backbone
2. -> RPN
3. -> NMS(no cls_score_threshold, nms_threshold=0.7)
4. -> bbox Branch
5. -> NMS(cls_score_threshold=0.05, nms_threshold=0.5)
6. -> mask Branch
7. -> plane clustering
8. -> NMS(cls_score_threshold, nms_threshold) -> final results

note:

• The step 1-6 are the same with Mask RCNN without changing thresholds.
• For PMTD, we add step 7 to calculate out quadrilaterals, and step 8 to filter the low score ones and NMS the near ones.
• The thresholds in step 8 are selected to achieve the best F-measure and can be viewed in the releasing code.

Q2: Have you do the ablation study on Data Augmentation, RPN Anchor and OHEM. In my experiments, Data Augmentation and OHEM improve the performance, but modification for RPN Anchor does not work.

A2: Sorry, we haven't done the ablation study on Data Augmentation. But for the modification on anchor, we believe that for the RPN-based methods which need setting the anchor manually, adjusting the anchor for the dataset is always no harm.

[jylins]

@JingChaoLiu Thanks for your tips 👍