Weight normalization - Githubissues

accosmin / nano

C++ library [machine learning & numerical optimization] - superseeded by libnano

MIT License

1 stars 0 forks source link

Weight normalization #164

Closed accosmin closed 6 years ago

accosmin commented 7 years ago

see: https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf
can implement new affine & conv3d layers that do this weighting
goal: may speed-up training

accosmin commented 7 years ago

alternative: try a parametrization of the form w = v / ||v||
alternative: w = exp(g) * v / ||v|| (like suggested in the paper)