Performance issue

[lxtGH]

Hi! Thanks for opensource such project for the public. I want to re-produce the mega-model results. The performance is about 1 map lower than the paper. Has anyone successfully reproduced the experiment results ?

Mega-R101: 82.0 Paper: 82.9 Mega-R50: 76.1 Paper: 77.3

The reuslt is best results from multiple experiments ?

[Scalsol]

Hi, what config did you use? If you use 4 gpus, the result of ResNet101 backbone should be close to 82.9. The results of 8 gpus may be lower. The result of ResNet50 backbone is not reported in paper and its performance is not stable, all models with ResNet 50 backbone suffer from the same issue (we mention this in the note). We report this result just for reference :)

[lxtGH]

Thanks for your reply. Mega-R101 uses 4 gpus(82.0, default config) when i use 8 gpus training, the reuslt is 81.5.

[lxtGH]

I will re-run default mega-r101-4gpus. @Scalsol By the way, why 8 gpu training results in lower results? Could it can be handled by introduced more training iteration?

[Scalsol]

I could not figure out why and more training iteration could not handle the problem. I haven't run many experiments with 8 gpus for at that time I have no enough GPU resources.

[lxtGH]

Report New results 82.4. Is this caused by initial seed problems?

[Scalsol]

Yes, the initial seed will influence the overall result (In maskrcnn_benchmark it is not fixed, unlike mmdetection). In my experiments the fluctuation is around 0.3 AP. As AP50 is not a so stable metric compared to mAP and the ImageNet VID dataset itself is not a good dataset, I think this is an acceptable result. If you have further question, you could find me out through WeChat: Scalsol

[lxtGH]

Thanks for your reply!