unstable gradient

[ih4cku]

vanishing and exploding gradient / sensitivity

• (must see) X. Glorot and Y. Bengio. Understanding the difficulty of trainingdeep feedforward neural networks. InAISTATS, 2010.
• (must see) Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." ICML (3) 28 (2013): 1310-1318.
• Why are deep neural networks hard to train?

[ih4cku]

weights initialization

• deepdish - Initialization of deep networks

[ih4cku]

Highway and Residual network

• original papers
• 干货 | ICCV 2015 揭开152层神经网络的面纱 1, 2
• Re-thinking Deep Residual Networks
• Highway networks are to deep residual networks what LSTMs are to ...?

following works:

• FractalNet
• Resnet in Resnet
• Deep Networks with Stochastic Depth