question about mAP for DOTA dataset

[teeinn]

I re-implemented your work for training Dota dataset but I got much lower mAP for dota test dataset.

* training & validation---------------------------------------------------------------------------------------------------------------

• dataset : DOTA - v1.0 (training, validation dataset -- all cropped by using your img_split.py)
• configuration : faster_rcnn_orpn_r101_fpn_1x_ms_rr_dota10.py
• parameters : not changed at all (same with your implementation)

* test-------------------------------------------------------------------------------------------------------------------------------------

• dataset : DOTA - v1.0 (test dataset -- all cropped by using your img_split.py)
• configuration : faster_rcnn_orpn_r101_fpn_1x_ms_rr_dota10.py
• used epoch 12 checkpoint
• sent to DOTA v 1.0 server - Evaluation of DOTA Task1 and got this result ---> mAP: 0.41229681978600863 ap of each class: plane:0.5396662332791309, baseball-diamond:0.2710892710892711, bridge:0.5154165616433038, ground-track-field:0.1484848484848485, small-vehicle:0.27250426663927574, large-vehicle:0.2622058637252267, ship:0.7108541529636384, tennis-court:0.5453417550191744, basketball-court:0.33560606060606063, storage-tank:0.09090909090909091, soccer-ball- field:0.22898154477101842, roundabout:0.19935158396696856, harbor:0.7828865404399028, swimming- pool:0.7302785521014554, helicopter:0.5508759711517627

According to your report, it should be 80.52 (backbone resnet101). What have I done wrong? What should I change? Waiting for your quick response! Thank you!

[jbwang1997]

Could you provide the number of trainval images and test images. The test set of the DOTA dataset consists of two zip files. I consider if you forget to unzip one of zip files.

[teeinn]

I think you are right. I thought I unzipped both of zip files (part1, part2). However, after cropping images, test images were 30087, which is smaller number than number of cropped validation images.

Before crop images ----> After img_split.py ----> After I fix train : 1411 ----> 52057 val : 458 ----> 35462 test : 937 ----> 30087 ----> 71888

now, it is 71888. Is this correct number of data images?

[jbwang1997]

Yes, the 71888 is the correct number of test set.

[teeinn]

Thank you! I will try one more time with this test set.

[colorfully-ybf]

Yes, the 71888 is the correct number of test set.

why is 71888? I have the same train val and test.After siplit,my picture is 10833 test and 12800 trainval.The parameter of gap is 200 and the pixel is 1024*1024.

[jbwang1997]

71888 images are generated by multiple-scale mode. 10833 images are generated by single-scale mode.

[colorfully-ybf]

Tanks

[fhaji75]

@teeinn When I'm running test.py on DOTA dataset I got mAP=0 with this comment : Result is Only for reference, final result is subject to DOTA_devkit

+--------------------+-----+--------+--------+--------+ | class | gts | dets | recall | ap | +--------------------+-----+--------+--------+--------+ | large-vehicle | 0 | 102855 | 0.0000 | 0.0000 | | swimming-pool | 0 | 12696 | 0.0000 | 0.0000 | | helicopter | 0 | 11973 | 0.0000 | 0.0000 | | bridge | 0 | 33974 | 0.0000 | 0.0000 | | plane | 0 | 19639 | 0.0000 | 0.0000 | | ship | 0 | 66079 | 0.0000 | 0.0000 | | soccer-ball-field | 0 | 6102 | 0.0000 | 0.0000 | | basketball-court | 0 | 4562 | 0.0000 | 0.0000 | | ground-track-field | 0 | 7147 | 0.0000 | 0.0000 | | small-vehicle | 0 | 126984 | 0.0000 | 0.0000 | | baseball-diamond | 0 | 7429 | 0.0000 | 0.0000 | | tennis-court | 0 | 9675 | 0.0000 | 0.0000 | | roundabout | 0 | 10242 | 0.0000 | 0.0000 | | storage-tank | 0 | 41839 | 0.0000 | 0.0000 | | harbor | 0 | 25446 | 0.0000 | 0.0000 | +--------------------+-----+--------+--------+--------+ | mAP | | | | 0.0000 | +--------------------+-----+--------+--------+--------+ I use the ss_test.json file for split test data, Is that correct? Do you know why this happened?