search

jessemelpolio / Faster_RCNN_for_DOTA

Code used for training Faster R-CNN on DOTA
https://arxiv.org/abs/1711.10398
Apache License 2.0
336 stars 102 forks source link

mAP becomes low after merging files. #26

Closed mamunir closed 5 years ago

mamunir commented 5 years ago

Checked resultsmerge and dota evaluation task 2 python files, they both seems legit. Is it normal, I do not think so. After merging, it should increase? On crop images, I have mAP=40 approx but after merging, I got mAP=15 approx.

I set source file in result merge to be 15 files destination file, where it should be saved after merge and nms

For dota evaluation task 2, detpath = r'PATH_TO_BECONFIGURED/Task1{:s}.txt' --> destination file, where it should be saved after merge and nms annopath = r'PATH_TO_BE_CONFIGURED/{:s}.txt' --> val/labelTxt imagesetfile = r'PATH_TO_BE_CONFIGURED/valset.txt' --> text file path for original large img size validation set

Please see my settings and comment. Thanks

@jessemelpolio @dingjiansw101

dingjiansw101 commented 5 years ago

It is not normal. I am not sure of the problem. As a suggestion, you can visualize the predicted bounding boxes on images(for both crop images and large size images), and see if it is correct.

mamunir commented 5 years ago

I have stopped after that day. Resumed it now. I have seen by drawing the boxes after merging on respective images. Boxes become wide as compared to cropped images. In cropped images, they were tighter but in original images, they become loose. I am looking into code again, how it handles the variations like translation and mapping etc. Let me know, if you understand the problem. This looks like weird problem

dingjiansw101 commented 5 years ago

What is the scale(0.5, 1 or 2?) you set when crop or merge images?

mamunir commented 5 years ago

rate is set to 1. I see in code, this refer to scale.

dingjiansw101 commented 5 years ago

Could you provide your results? And give a detail operation description. If so, I will try to reproduce your results.

mamunir commented 5 years ago

sure. am i supposed to send on jding[at]whu[dot]edu[dot]cn?

dingjiansw101 commented 5 years ago

Yes, and attach a detailed description of your operation.

mamunir commented 5 years ago

okay. thank you for your time

mamunir commented 5 years ago

sent, please check your email

dingjiansw101 commented 5 years ago

@mamunir I did not receive it yet. You can try jianding101@gmail.com

mamunir commented 5 years ago

maybe in spam. sent on gmail also.

mamunir commented 5 years ago

as a general question, mAP always lower even a little when merged or always higher than cropped when merged? or random?

dingjiansw101 commented 5 years ago

It seems that you use oriented labels. However, to calculate mAP for Task2, we need to use the horizontal labels. We have uploaded the Task2_gt in Baidu drive before. Now we also add the Task2_gt in Google drive. You should use Task2 version.

mamunir commented 5 years ago

Yes you are right, problem lies here. I also sorted it just, see your input now. What I did, make min max of x points and y points out of oriented labels. Now from 40.25 mAP, it degrades to 38.5 rather than 15.9 previously. This seems normal to me, good?

dingjiansw101 commented 5 years ago

I have not tested on crops before. But in my opinion, the difference between crops and large images is caused by the instances on the edge. There may be only part of an instance appear on crops. In the generated ground truth of crops, if the proportion is below 0.7, we give a difficult label. You can try to ajust the paramater or more models and verify relationship between crop results and large image results.

mamunir commented 5 years ago

Thank you :)