Weight initializations

[dmccloskey]

Description

w=np.random.randn(layer_size[l],layer_size[l-1])*np.sqrt(2/layer_size[l-1])

Objectives

• Xavier initializations for Non-ReLU FFNN (Norm Dist std dev = sqrt(2. / (in + out)))
• He et al, 2015 initialization for ReLU FFNN (Norm Dist std dev = sqrt(2/in)

References

• Xavier initialization: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
• He et al, 2015: https://arxiv.org/pdf/1502.01852.pdf
• https://towardsdatascience.com/random-initialization-for-neural-networks-a-thing-of-the-past-bfcdd806bf9e