search

danpovey / pocolm

Small language toolkit for creation, interpolation and pruning of ARPA language models
Other
90 stars 48 forks source link

target size / initial-threshold #53

Closed vince62s closed 8 years ago

vince62s commented 8 years ago

Dan, Currently the way it is thought is not optimal. we need to have a "proper" initial-threshold so that the prunelm script works. if the initial threshold is too high some target size may not work. is the current WIP supposed to fix this ?

thanks

wantee commented 8 years ago

No, the current WIP will also die if we get a lm whose size is less than target size after pruning with the initial threshold. I think the reason is that, in principle, we wish to prune from a larger LM and gradually approach the target size. Because once a LM is smaller than target size, we can't prune from it in the next iteration, instead, we have to go back to a larger LM in history. This would be a waste of time. However, maybe we could try to use another model to guess the initial threshold at the beginning, if user didn't specify a initial threshold. Dan, do you think that is feasible?

danpovey commented 8 years ago

It would definitely be possible to modify the code so that if the initial threshold is too high, instead of dying it divides the initial threshold by 4 and starts again. Dan

On Thu, Aug 18, 2016 at 8:51 AM, Wang Jian notifications@github.com wrote:

No, the current WIP will also die if we get a lm whose size is less than target size after pruning with the initial threshold. I think the reason is that, in principle, we wish to prune from a larger LM and gradually approach the target size. Because once a LM is smaller than target size, we can't prune from it in the next iteration, instead, we have to go back to a larger LM in history. This would be a waste of time. However, maybe we could try to use another model to guess the initial threshold at the beginning, if user didn't specify a initial threshold. Dan, do you think that is feasible?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/53#issuecomment-240767682, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu31WdDdQeNnPZAePVLYSF0bqgEqbks5qhH9-gaJpZM4Jnb4q .

wantee commented 8 years ago

Modified the code to automatically adjust initial threshold.

BTW, Dan, do you see my comment about prev_change_power in the WIP PR?

danpovey commented 8 years ago

Regarding your question "Do we need to relax the limitation of prev_change_power?", maybe you can relax it a bit, e.g. to 1.5, but I think the real fix for this is that if backtracking was needed, then before applying the model again to work out the new num-xgrams, we prune again with the same num-xgrams as last time. This issue of where the model can change more on this iter than on the previous iter, is I think something that happens only when we have just increased the threshold a lot for a fairly high-order model, when it needs to prune again to see the full effect of the threshold (due to "protected" n-grams, that cannot be pruned until we've pruned away the state they lead to).

On Thu, Aug 18, 2016 at 10:22 PM, Wang Jian notifications@github.com wrote:

Modified the code to automatically adjust initial threshold.

BTW, Dan, do you see my comment https://github.com/danpovey/pocolm/pull/45#issuecomment-240322703 about prev_change_power in the WIP PR?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/danpovey/pocolm/issues/53#issuecomment-240918063, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu9S14F2wVZ3MDDkcYg0INpdKvB50ks5qhSFlgaJpZM4Jnb4q .