Pretrained language models to download

kpu / kenlm

KenLM: Faster and Smaller Language Model Queries

http://kheafield.com/code/kenlm/

Other

2.5k stars 513 forks source link

Pretrained language models to download #276

Closed bjascob closed 4 years ago

bjascob commented 4 years ago

I have some python code that uses KenLM, but it doesn't include the actual model. Is there somewhere I can download a trained US English model that's reasonable quality/size (ie.. a few GB or less). I've Googled for this but haven't found what I'm looking for.

kpu commented 4 years ago

Here's one. http://www.statmt.org/moses/RELEASE-4.0/models/es-en/lm/europarl.lm.3 and also a binary format: http://www.statmt.org/moses/RELEASE-4.0/models/es-en/lm/europarl.binlm.3 . But that doesn't mean very much since you really want a model preprocessed the same way as your other text, hence the lack of prebuilt models.