LM experiments - Githubissues

ybisk / HMM-RNN

2 stars 0 forks source link

Closed janmbuys closed 6 years ago

janmbuys commented 6 years ago

Perform sanity checks that models behave roughly as before with updated implementation.
Add dropout and do a minimal amount of hyperparameter tuning (although for good LM performance better optimization techniques will required).
Run experiments to compare models on PTB setup (once available).

ybisk commented 6 years ago

Sanity check seems fine. Currently running w/ hidden_dim/clusters == 50/100 and HMM w/ and w/out 1-hot embeddings

ybisk commented 6 years ago

Rerunning experiments with step fixed

janmbuys commented 6 years ago

The val numbers look reasonable, expect for HMM, where I am pretty sure there has to be a bug.

janmbuys commented 6 years ago

This should approximately replicate the pytorch LM baseline:

python ptb_main.py --type lstm --optim sgd --lr 20 --dropout 0.2 --tie-embeddings --hidden-dim 200 --embed-dim 200 --initrange 0.1

janmbuys commented 6 years ago

Pushed a possible HMM bug fix.