search

casperkaae / parmesan

Variational and semi-supervised neural network toppings for Lasagne
Other
208 stars 31 forks source link

[WIP] Weight Normalization layer #33

Closed skaae closed 8 years ago

skaae commented 8 years ago

preliminary implementation of http://arxiv.org/abs/1602.07868

This is an implementation of equation 1+2 in the paper. Please check if that I didn't take norm over the wrong axis :)

Should we support mean--only batch normalization? In that case i think the easiest is to create mean-only batch norm layer and wrap the weight normalization layer.

Maybe we should provide a helper function for mean-only bn + WN

wuaalb commented 8 years ago

Nice!

Have you tried this for some generative task? From skimming through the paper it seems it only compares WN w/ Polyak averaging (and norm bounding?) with no normalization at all. It would be nice to also compare to BN and WN + mean-only BN, and maybe without the averaging.