Closed sofsoo1995 closed 6 years ago
I've downloaded the pretrained model from web and ran it again using 8 gpus, got 72.9. Have you tried other pretrained model? Did they get the right result?
Thx you for your answers, The results I'm reporting is with the pretrained model and not my training. I only have tried res101.384x288 and res50.256x192 and the results are approximately the same(I've just reported res101.384x288). qualitatively, the model works on simple cases but it does a lot of errors for more complicated cases. I will try the other models and will also try to train(with 2 Titan X and 1 GTX 1080 )
Do you check the validation dataset?
minival dataset: https://drive.google.com/drive/folders/15loPFQCMQnJqLK1viSMeIwTFT-KbNzdG
minival det: https://drive.google.com/drive/folders/1BllF9--dN9uV3FRROcmuIbwNCcn7cCP0?usp=sharing
Ok, I've re downloaded pre trained model and I've re tested everything. and actually res50.256x192 has good results (as it is indicated) ! However the other methods failed (res50.384x288, and res101.384x288). So the problem might come from the input size. Is it possible that it is a problem of hardware(because I only use 3 GPUs) ?
I put the details here(with pretrained model):
res50.256x192
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.697 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.883 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.770 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.662 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.761 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.764 Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.927 Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.823 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.715 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.830
res50.384x288
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.206 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.459 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.162 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.170 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.268 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.310 Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.591 Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.280 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.250 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.392
res101.384x288
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.430 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.682 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.458 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.385 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.511 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.534 Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.780 Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.566 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.463 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.631
I think it can't be the problem of hardware. Since it's worse in 384x288 size, did you run the pre-trained model in the corresponding model folder?
I found the mistake. I modified config and with silly copy-pastes I put the wrong i input size in the two networks. My Bad !
However, I still don't understand why is there this huge difference of results if I put a wrong input size for testing.
But thank you very much for helping me !
Congratulation for your COCO Challenge result, and thank you for sharing your code.
I'm testing your code and the problem is I have these results on the validation set of 2014:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.430 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.682 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.458 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.385 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.511 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 20 ] = 0.534 Average Recall (AR) @[ IoU=0.50 | area= all | maxDets= 20 ] = 0.780 Average Recall (AR) @[ IoU=0.75 | area= all | maxDets= 20 ] = 0.566 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.463 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.631
-I'm using ResNet-101 with an input size of 384x288 -I haven't almost changed anything in the code(except the config file, in mptest I put dump_method=1 as argument for the function MultiProc). -I'm using the pretrained model you have trained. -Also, for the dataset, I have downloaded the 2014 version(train and val). -I'm using the annotation file and also the bounding box you've given. -I don't understand why is there a huge difference between your results and mine ! Have I done something wrong ?