Batch normalization between input and activation across all examples in a minibatch. Need to implement a moving average for the averages and variances (0.9 prev + 0.1 cur), so that at test time, the moving average and variance can be used.
Objectives
[ ] BatchNormActivation method
[ ] with tests
[ ] addBatchNorm to ModelBuilder
[ ] with tests
References
Ioffe et al, 2015 Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167v3
Description
Batch normalization between input and activation across all examples in a minibatch. Need to implement a moving average for the averages and variances (0.9 prev + 0.1 cur), so that at test time, the moving average and variance can be used.
Objectives
addBatchNormto ModelBuilderReferences