Open waleedka opened 7 years ago
Thanks, this is definitely a discrepancy, likely born of the fact that I threw this layer together in a couple minutes right after seeing the paper. Also because cuDNN's batchnorm does the scale+shift automatically. If you want to submit a pull request to fix it (just moving the gamma+beta outside the cuDNN bnorm call would do it) feel free, though I'm not really supporting much of the code in this repo thse days <_<
I came across your implementation of batch re-normalization in the BatchReNormDNNLayer class, and I think there might be an error that might be affecting the model's performance.
My understanding of batch re-norm is that it applies the standard BN normalization first, then applies the r/d correction, and then finally applies the gamma/beta scaling and bias. Something along the lines of this:
However, this line is applying the r/d correction after the scaling and centering with gamma and beta. https://github.com/ajbrock/Neural-Photo-Editor/blob/master/layers.py#L128
It probably works anyway, based on the good results you seem to have gotten. I just thought I'd bring it to your attention.