search

danpovey / pocolm

Small language toolkit for creation, interpolation and pruning of ARPA language models
Other
90 stars 48 forks source link

Crash in float-counts-prune #107

Open francisr opened 2 years ago

francisr commented 2 years ago 
float-counts-prune: float-counts-prune.cc:362: float pocolm::FloatCountsPruner::PruningLogprobChange(float, float, float, float): Assertion `count > 0.0 && discount > 0.0 && backoff_total > backoff_count && backoff_total >= 0.99 * discount' failed.

More precisely, it happens because backoff_total == backoff_count.
I'm not familiar with the code, would it be problematic to have equality?
I think it's happening because I've got the same token occuring at the end of each sentence, but that hasn't been a problem before.

danpovey commented 2 years ago

Can you change that > to >= and see if it seems to work normally? I don't see an immediate/obvious reason why that needs to be > and not >=.