troubling in Reproduce paper AP

[Google1234]

Hi，thank you for your great work ，We tried to reproduce the paper indicators, but encountered problems： This is the original paper indicator： COCO（Not using multi-scale test） Method | Data | Backbone | AP | AP50 | AP75 | APS | APM | APL RefineDet512（paper） | trainval35k | ResNet-101 | 36.4 | 57.5 | 39.5 | 16.6 | 39.9 | 51.4

This is our reproduce result: RefineDet512（reproduce）| trainval35k | ResNet-101 | 32.3 | 48.7 | 35.7 | 13.5 | 36.1 | 48.5 We using P100 , 8 GPUs , batch size 64 , learing rate 0.02 (200K) , 0.002(200K) ,0.0002(200k) The results of the recurrence did not reach the original paper, and we also tried it many times but still failed. If there are other skills ，we need pay attention to? THX

[sfzhang15]

@Google1234 Hi, you use P100 , 8 GPUs , batch size 64，since P100 has 16G memory, does it means that every GPU has 4 images?

[Google1234]

every GPU has 8 images , 8 GPUs , total batch size is 64

[sfzhang15]

@Google1234 Which framework do you use? Pytorch? Besides, the learning rate setting is different from ours. We use 0.001 for 400k, 0.0001 for another 80k and 0.00001 for the last 60k. And we use 4 24G GPUs and each GPU has 5 images.

[Google1234]

using pytorch Why batch size is only 4 for single GPU？ our batch size can reach 8 for single GPU with 16G memory，image input size is 512*512

[sfzhang15]

@Google1234 If you use PyTorch, you can put more images in one GPU than Caffe. If using Caffe, a 16G GPU can input 4 512x512 images with resnet101 and a 24G GPU can input 5 512x512 images with resnet101.

[Google1234]

Got it ! So Is there any other skill ，we need pay attention to?

[sfzhang15]

@Google1234 Since the limitation of Caffe and GPU memory, we have to use batch size=20 with 4 GPUs to train RefineDet512_Res101. If you can put 8 images per GPU, you'd better use batch size=32 with 4 GPUs and the learning rate schedule is the same as SSD_COCO. If you use batch size=64 with 8 GPUs, maybe the learning rate schedule need to be adjusted.