Hi,
I am trying to build and interpolate very small language models (most higher
order n-grams are unique). I am not able to interpolate ARPAs, because it
always throw the following error:
$ estimate-ngram -order 3 -text 1.txt -s ML -wl 1.arpa
...
$ estimate-ngram -order 3 -text 2.txt -s ML -wl 2.arpa
...
$ interpolate-ngram -order 3 -lm "1.arpa, 2.arpa" -wl 3.arpa
...
interpolate-ngram: src/InterpolatedNgramLM.cpp:327: void
mitlm::InterpolatedNgramLM::_EstimateBows(): Assertion `!anyTrue(isnan(bows))'
failed.
Aborted (core dumped)
PS 1. The same when doing open vocab (-unk)
PS 2. My minimalistic arpa do have most of BOW set to -99
PS 3. There are quite a lot of n-grams with -log(p) == 0.00000 in arpa
PS 4. I found out that "</s>" 1-gram _DOES_NOT_ have BOW in arpa
PS 5. I am using mitlm-0.4.1
Any ideas?
Original issue reported on code.google.com by cypre...@gmail.com on 22 Jul 2015 at 8:03
Original issue reported on code.google.com by
cypre...@gmail.com
on 22 Jul 2015 at 8:03