Perplexity including OOVs

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries

http://kheafield.com/code/kenlm/

Other

2.45k stars 512 forks source link

Open rnajim opened 2 years ago

rnajim commented 2 years ago

Hello, is there any scripts or parameter that I can add to get the total perplexity including/excluding OOVs?

I want to find a way to calculate the perplexity including OOVs

RajGothi commented 10 months ago

Run query command. You will get both perplexity including/excluding OOVs.