PTB LM setup

[janmbuys]

• Load PTB data.
• Compute perplexity on training and validation data.
• Train with truncated backpropagation through time.

[janmbuys]

Update: Implemented the above (as ptb_main.py).

However, we still need to implement standard regularization and optimization techniques to get reasonable results:

• First implement Zaremba et al 2014.
• Later consider porting AWD-LM.

[janmbuys]

Note that LSTM regularization techniques might not work for HMM as due to the hidden state bottleneck it will have a harder time memorizing the training data.

[ybisk]

I hope there's an error in the perplexity?

[janmbuys]

Update: Implemented dropout and basic SGD optimization strategies.