Can not reproduce the results

[gaopeng-eugene]

I run the code model=data/models/caffenet_fast_rcnn_iter_40000.t7 ./scripts/eval_fastrcnn_voc2007.sh get: Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.230 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.492 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.189 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.015 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.112 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.306 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.263 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.355 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.364 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.080 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.257 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.441

{ year : "2007" test_nsamples : -1 test_augment : false proposals : "selective_search" test_model : "data/models/caffenet_fast_rcnn_iter_40000.t7" max_size : 1000 test_best_proposals_number : 2000 test_save_res : "" test_add_nosoftmax : false test_save_raw : "" disable_memory_efficient_forward : false test_data_offset : -1 proposal_dir : "./data/proposals" test_save_res_prefix : "" test_bbox_voting_nms_threshold : 0.5 test_num_iterative_loc : 1 scale : 600 dataset : "pascal" test_min_proposal_size : 2 test_nGPU : 8 test_load_aboxes : "" test_bbox_voting : false test_just_save_boxes : false transformer : "RossTransformer" test_bbox_voting_score_pow : 1 test_set : "test" test_use_rbox_scores : false test_nms_threshold : 0.3 } dataset: pascal_test2007 proposals_path: { 1 : "/data/scratch/gaop/deep_learning_detection/multipathnet/data/proposals/VOC2007/selective_search/test.t7" }

[gaopeng-eugene]

run the code model=data/models/vgg_fast_rcnn_iter_40000.t7 ./scripts/eval_fastrcnn_voc2007.sh get Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.304 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.599 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.277 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.032 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.175 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.390 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.304 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.414 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.421 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.317 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.503

opt { year : "2007" test_nsamples : -1 test_augment : false proposals : "selective_search" test_model : "data/models/vgg16_fast_rcnn_iter_40000.t7" max_size : 1000 test_best_proposals_number : 2000 test_save_res : "" test_add_nosoftmax : false test_save_raw : "" disable_memory_efficient_forward : false test_data_offset : -1 proposal_dir : "./data/proposals" test_save_res_prefix : "" test_bbox_voting_nms_threshold : 0.5 test_num_iterative_loc : 1 scale : 600 dataset : "pascal" test_min_proposal_size : 2 test_nGPU : 8 test_load_aboxes : "" test_bbox_voting : false test_just_save_boxes : false transformer : "RossTransformer" test_bbox_voting_score_pow : 1 test_set : "test" test_use_rbox_scores : false test_nms_threshold : 0.3 } dataset: pascal_test2007 proposals_path: { 1 : "/data/scratch/gaop/deep_learning_detection/multipathnet/data/proposals/VOC2007/selective_search/test.t7" }

[gaopeng-eugene]

run test_nGPU=4 test_nsamples=5000 ./scripts/eval_coco.sh result Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.077 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.237 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.031 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.079 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.136 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.099 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.138 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.141 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.161 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.239

{ year : "2014" test_nsamples : 40000 test_augment : false proposals : "sharpmask" test_model : "./data/models/resnet18_integral_coco.t7" max_size : 1000 test_best_proposals_number : 400 test_save_res : "" test_add_nosoftmax : false test_save_raw : "" disable_memory_efficient_forward : false test_data_offset : -1 proposal_dir : "./data/proposals" test_save_res_prefix : "" test_bbox_voting_nms_threshold : 0.5 test_num_iterative_loc : 1 scale : 800 dataset : "coco" test_min_proposal_size : 2 test_nGPU : 8 test_load_aboxes : "" test_bbox_voting : false test_just_save_boxes : false transformer : "ImagenetTransformer" test_bbox_voting_score_pow : 1 test_set : "val" test_use_rbox_scores : false test_nms_threshold : 0.3 } dataset: coco_val2014
proposals_path: { 1 : "/data/scratch/gaop/deep_learning_detection/multipathnet/data/proposals/coco/sharpmask/val.t7" }

[szagoruyko]

can't reproduce, unfortunately

[XiongweiWu]

@szagoruyko I cannot get the expected result as well. So I wonder whether you can share your log file of config?

[szagoruyko]

@XiongweiWu can you run test.lua and see if it passes? I used latest torch and a clean clone of the code, and got the expected results

[XiongweiWu]

@szagoruyko quite strange.....When I train the alexnet detection network by myself according to your script, the detection result is still around 50%....

[szagoruyko]

@XiongweiWu can you post your log?

[XiongweiWu]

@szagoruyko https://gist.githubusercontent.com/XiongweiWu/ba81e45f363f489e837714e723dd7ed8/raw/5481b48b4eee3dbb8ba0fa4303e9738d835bf011/multipath-log Can you access the log file?

[Hokie23]

After complete training of frcnn using the vgg model I am getting a mAP of .49. Is the coco evaluation different from frcnn evaluation?

[szagoruyko]

I updated pycocotools and reproduced the issue, some of the commits here https://github.com/pdollar/coco/commits/master/PythonAPI/pycocotools/coco.py caused this cc @tylin . We will track it down, meantime use my fork https://github.com/szagoruyko/coco/tree/load-numpy

[chrieke]

Seems like this issue still exists. Running it with the newest pycocotools version: test_nGPU=4 test_nsamples=5000 ./scripts/eval_coco.sh

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.072 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.221 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.031 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.016 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.077 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.132 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.095 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.135 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.137 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.029 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.156 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.231

Which is pretty similar to gaopeng-eugene's last post.

[gcheron]

Thanks @szagoruyko ! I had the same issue but changing to your load-numpy branch solved it! Do you have any idea why original coco repo does not reproduce?