akhilpm
commented
1 month ago Hi Li,
The threshold of 0.85 might be because I pushed a version when I was experimenting with a different set of values.
When using Faster RCNN, the difference in gain for overall mAP is not that big in my experience too (I got something in the range of 0.3-0.5). However, the small objects AP is improved; You can also see that when you use 0.7 the APs is 26.19 and when you use 1 (no crops) the APs is 25.61.
However, the FCOS detector had a different profile. For FCOS, the overall mAP showed a difference of 1.2-1.4 when the threshold was set properly. As you can see from the tables in the paper, FCOS performs better than Faster RCNN.
I think Faster RCNN is already filtering ambiguous small object detection in its two-stage process after training with augmented crops. FCOS on the other hand deals with many more candidate regions before the filtering (the NMS), so the detection from scaled crops is helping it with more confident small object detection.
In the paper, it mentions that a confidence threshold of 0.7 was used for the CZ Detector to obtain 1-3 crops per image for a fair comparison. However, in your provided code, I noticed that the confidence threshold is set to 0.85 in the rcnn_crop.py configuration. In particular, I noticed that when the confidence threshold is set to 1 during evaluation, the AP results are nearly identical to those obtained with a threshold of 0.85, despite no cropped regions being processed when the threshold is set to 1. This outcome raised some questions for me regarding the behavior of the detector under these conditions. Could you please clarify why this happens, and whether this is expected behavior when the confidence threshold is set to 1? I would also appreciate any insights into why 0.85 was chosen as the default threshold in rcnn_crop.py instead of 0.7. For more details, during my testing, I used the following command for training: python train_net.py \ --num-gpus 1 \ --config-file configs/RCNN-FPN-CROP.yaml \ OUTPUT_DIR outputs_FPN_CROP_VisDrone For evaluation, I modified the confidence threshold but did not change any training-related parameters. The evaluation command was: python train_net.py --eval-only --num-gpus 1 --config-file configs/RCNN-FPN-CROP.yaml MODEL.WEIGHTS model_0089999.pth
I noticed that the confidence threshold is set to 0.85 in the rcnn_crop.py configuration. (crop_boxes = crop_boxes[crop_boxes.scores>0.85])
Here’s the result of evaluation using a confidence threshold of 0.7:
Here’s the result of evaluation using a confidence threshold of 0.85:
Here’s the result of evaluation using a confidence threshold of 1: