search

kazuto1011 / deeplab-pytorch

PyTorch re-implementation of DeepLab v2 on COCO-Stuff / PASCAL VOC datasets
MIT License
1.09k stars 282 forks source link

BatchNorm Momentum Setting in Pytorch #79

Closed yuhuan-wu closed 4 years ago

yuhuan-wu commented 4 years ago

Hello kazuto! An awesome job for re-implenting deeplab-v2 in Pytorch! I find a BN typo in this repo, although such typo will not affect the performance, as people will not train BN layers as default.

In Pytorch , the momentum parmameter of nn.BatchNorm2d class is 0.1. In fact, this momentum param is actually 1 - true_momentum, and the true momentum is 0.9 as default. It can refer to the documentation of Pytorch 0.4.1 torch.nn.BatchNorm2d.

However, in your code, libs/models/resnet.py, 

self.add_module("bn", _BATCH_NORM(out_ch, eps=1e-5, momentum=0.999))

I know you wanna set the true momentum to be 0.999 to avoid the unstabitily of the mean value/std caused by small batchsize. I think it should be set as a right value: 0.001 and it will work well :).

kazuto1011 commented 4 years ago

Thank you for pointing it out! I will fix all the related lines.