RNNs papers to read

[ih4cku]

ablation studies of gates

• Chung, Junyoung, et al. "Empirical evaluation of gated recurrent neural networks on sequence modeling." arXiv preprint arXiv:1412.3555 (2014).
• Greff, Klaus, et al. "LSTM: A search space odyssey." arXiv preprint arXiv:1503.04069 (2015).

[ih4cku]

Make implementation fast

• Baidu Silicon Valley AI Lab http://svail.github.io/

[ih4cku]

vanishing and exploding gradient / sensitivity

• (must see) Pascanu, Razvan, Tomas Mikolov, and Yoshua Bengio. "On the difficulty of training recurrent neural networks." ICML (3) 28 (2013): 1310-1318.
• Written Memories: Understanding, Deriving and Extending the LSTM
• Why are deep neural networks hard to train?
• Why can Constant Error Carousels (CECs) prevent LSTM from the problems of vanishing/exploding gradients?

[ih4cku]

BPTT

• Section 2.8.6 of Ilya Sutskever’s Ph.D. thesis
• Styles of Truncated Backpropagation