search

yandex / faster-rnnlm

Faster Recurrent Neural Network Language Modeling Toolkit with Noise Contrastive Estimation and Hierarchical Softmax
Other
562 stars 138 forks source link

Benchmark on 1 Billion #7

Open sherjilozair opened 9 years ago

sherjilozair commented 9 years ago

You report training times in the 1 Billion dataset, but not test perplexity. It would be helpful to compare this implementation with others if you could post the test perplexity and the hyperparameters that could be used to reproduce the result.

Also, thanks for putting this on Github. :)

akhti commented 9 years ago

I will definitely add some ASAP! And could you advice any other implementations that could handle one billion word dataset? I'm awawre about torch-based HS. Everything else seems to be too slow for such a huge vocabulary.

sherjilozair commented 9 years ago

Possibly the only library that comes close to handling it right now is: https://github.com/nicholas-leonard/dp/.

Looking forward to your results.

akhti commented 9 years ago

I've finally added information about finally of different models. Check out https://github.com/yandex/faster-rnnlm#experiments

sherjilozair commented 9 years ago

The benchmarks are really cool! Thanks!

How do you interpolate with the KN-5 model, though? And from where is the KN-5 model obtained?

akhti commented 9 years ago

The model is downloaded from benchmark's repo. I used quite straightforward interpolation, i.e. mean per-word probability, as it's describe in the benchmarks' paper.