search

originrose / cortex

Machine learning in Clojure
Eclipse Public License 1.0
1.27k stars 111 forks source link

Weight normalization upgrade. #78

Open cnuernber opened 7 years ago

cnuernber commented 7 years ago

https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf

Need someone to get into this, test it, see where it works and where it does not.

cnuernber commented 7 years ago

Current example of weight initialization by hand to allow testing of different strategies:

https://github.com/thinktopic/cortex/blob/master/test/clj/cortex/nn/network_test.clj#L10

CalderBot commented 7 years ago

167 - As things stand after this PR, weight initialization for conv->relu defaults to the values recommended in He et. al. http://arxiv.org/abs/1502.01852, everything else defaults to Xavier (which is the same as He but with 1/2 the Gaussian width). One can specify initialization type explicitly in a layer with the argument syntax ":weights {:initialization {:type desired-type}}"

charlesg3 commented 7 years ago

I think there's a confusion here between weight initialization and weight normalization, re-opening issue.