search

isht7 / pytorch-deeplab-resnet

DeepLab resnet v2 model in pytorch
MIT License
602 stars 118 forks source link

Frozen the statistics of BN? #39

Closed nihaomiao closed 6 years ago

nihaomiao commented 6 years ago

Hello, do you implement the "frozen BN layers"? I find that running means still keep changing during training.

isht8 commented 6 years ago

running mean and variance would be constant because we use eval mode during training. (see here) The parameters of the batchnorm layer have no gradient because of this setting everywhere for batchnorm in model definition

nihaomiao commented 6 years ago

Hello, setting model.eval() during training can fix running means and vars. But now I am a bit confused since this setting can also affect other layers. Do other layers can be trained and updated in setting "model.eval()"?

isht7 commented 6 years ago

model.eval() only affects Batchnorm and dropout layer. Read more about this in pytorch docs. model.eval() is safe here because, there is no dropout layer in our model.

nihaomiao commented 6 years ago

Thank you! Now I know that model.eval() doesn't turn off the history tracking..... Thanks a lot!

isht7 commented 6 years ago

to stop pytorch from storing compute graph, use torch.no_grad()