search

jayurbain / mitlm

Automatically exported from code.google.com/p/mitlm
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Interpolation assertion `!anyTrue(isnan(bows))' failed. #50

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Hi,

I am trying to build and interpolate very small language models (most higher 
order n-grams are unique). I am not able to interpolate ARPAs, because it 
always throw the following error:

$ estimate-ngram -order 3 -text 1.txt -s ML -wl 1.arpa
...
$ estimate-ngram -order 3 -text 2.txt -s ML -wl 2.arpa
...
$ interpolate-ngram -order 3 -lm "1.arpa, 2.arpa" -wl 3.arpa
...
interpolate-ngram: src/InterpolatedNgramLM.cpp:327: void 
mitlm::InterpolatedNgramLM::_EstimateBows(): Assertion `!anyTrue(isnan(bows))' 
failed.
Aborted (core dumped)

PS 1. The same when doing open vocab (-unk) 
PS 2. My minimalistic arpa do have most of BOW set to -99
PS 3. There are quite a lot of n-grams with -log(p) == 0.00000 in arpa
PS 4. I found out that "</s>" 1-gram _DOES_NOT_ have BOW in arpa
PS 5. I am using mitlm-0.4.1

Any ideas?

Original issue reported on code.google.com by cypre...@gmail.com on 22 Jul 2015 at 8:03