search

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries
http://kheafield.com/code/kenlm/
Other
2.5k stars 513 forks source link

Entropy #370

Open i55code opened 2 years ago

i55code commented 2 years ago

Hi, is there a function call to calculate entropy directly from KenLM? What is the maths formula of model.score ? Is it the log probability or probability?

Is it true that in your code:

entropy(sent) = - model.score(sent) / length(sent) ? perplexity (sent) = 10 ^ (- model.score(sent) / length(sent) )?

Thanks!