search

jay-mahadeokar / pynetbuilder

pyNetBuilder is a modular pytonic interface with builtin modules for generating popular caffe prototxt network file definitions.
BSD 2-Clause "Simplified" License
328 stars 140 forks source link

Should ResNet work without Batch Norm layer? #8

Closed mrgloom closed 7 years ago

mrgloom commented 7 years ago

Should ResNet work without Batch Norm layer?

jay-mahadeokar commented 7 years ago

Not sure it can. BatchNorm layer gives you a lot of advantages, including regularization, solves problem of internal covariate shift and diminishing gradients which are very important once you go deeper. It also helps to train faster, with bigger learning rates, which is necessary when you are training such deep networks. More details are presented in Batchnorm paper.