young-jon
commented
7 years ago here are some links that might help http://ruishu.io/2016/12/27/batchnorm/ http://r2rt.com/implementing-batch-normalization-in-tensorflow.html
Open young-jon opened 7 years ago
young-jon
commented
7 years ago here are some links that might help http://ruishu.io/2016/12/27/batchnorm/ http://r2rt.com/implementing-batch-normalization-in-tensorflow.html
The code in dnn.py for batch normalization (as contributed by Lujia) does not seem to be complete. Population mean and variance should be used during test time. As far as I can tell, this isn't implemented, which could lead to incorrect predictions at test time. Does this mean that this functionality was never tested prior to pull request? It also seems that batch normalization isn't performed on the 1st hidden layer.
My guess is that the current validation and test functions are using a mean and variance that is similar enough to the population mean and variance to be almost unnoticeable, and therefore giving good accuracies/cost. However, try testing on just 2 extreme examples (such as the zero vector) instead of an entire set of data and I bet the predictions won't make sense.