Perplexity measurements are a useful way to evaluate the quality of a language model.
measure_perplexity.lua calculates the cross entropy and perplexity of a model (loaded from a checkpoint) relative to some test corpus. The calculations are done at both the character and the word level.
I think this script (or some similar script) could be useful for anyone who is interested in quantitatively comparing or ranking probabilistic language models.
Perplexity measurements are a useful way to evaluate the quality of a language model.
measure_perplexity.luacalculates the cross entropy and perplexity of a model (loaded from a checkpoint) relative to some test corpus. The calculations are done at both the character and the word level.I think this script (or some similar script) could be useful for anyone who is interested in quantitatively comparing or ranking probabilistic language models.