Open christianbuck opened 11 years ago
Where did you get the recipe from?
I tried 1/sqrt(fan_in) [1] and it performs better: for batchsize 4, the xor works most of the times, for batchsize 1 not so much. It was sqrt(6)/sqrt(fin_in+fan_out) before.
I pushed it, so you can try.
[1] http://www.willamette.edu/~gorr/classes/cs449/precond.html
I think I got this from Andrew Ngs examples but I am not sure. What about the hidden2output weights?
The cuda-convnet people seem to just use very small numbers and drop the dependence on layer size: http://code.google.com/p/cuda-convnet/wiki/LayerParams
Initial weights should probably depend on the activation function used in the hiden layer. Also we switched from standard sigmoid, 1/(1+exp(-x)), to tanh but weights are still initialized using a sigmoid receipe