search

jessemelpolio / Faster_RCNN_for_DOTA

Code used for training Faster R-CNN on DOTA
https://arxiv.org/abs/1711.10398
Apache License 2.0
336 stars 102 forks source link

Questions on the test mAP #5

Closed huangh12 closed 6 years ago

huangh12 commented 6 years ago

Hello, Thanks for your baseline code. I recently try to eval the mAP metric on validation dataset using your code with your trained model(rcnn_DOTA_quadrangle-0059.params). However, the output mAP is only ~0.1498, which is much lower than the numbes claimed in DOTA website. Here is my output details. From the 'check fp', it seems there are lots of false postive... Anyway, this mAP seems unnormal. But I try to visualize the prediction results, which however seems not so much bad. Could do please help?

default

dingjiansw101 commented 6 years ago

Hi, it seems that you did not do the nms. And is the result of the split images or the merged images?

huangh12 commented 6 years ago

Hello, sorry to reply late. I have tried to eval the results on merged images, the mAP is ~0.234. Details below:

('map:', 0.2347648790252074) ('classaps: ', array([ 24.94999407, 21.07866932, 23.07829282, 13.74383206, 24.00352528, 24.59458741, 20.74192156, 22.56024636, 17.0423561 , 38.00409317, 21.15414876, 28.11508966, 20.23769423, 33.62597449, 19.21689325]))

I think the nms is unnecessary when calculate the mAP, right?

dingjiansw101 commented 6 years ago

I think the nms is necessary. It would generate many FPs(false negative) if you do not do the nms.

jinbooooom commented 6 years ago

@huangh12 How do you detect and visualize?

huangh12 commented 6 years ago

@jinbooooom Follow the instruction in readme.md to test. During the test, they already have flag draw in pred_eval_dota_quadrangle function. Set it to true to visualize. What's more, note the nms has been uncommented by default. So one may need to implement their own nms function for quadrangle or utilize the existing function at here. The claimed number can be reproduced after doing the above.