There is a feature that I need for Kaldi reasons-- it will allow us to model <unk> with a phone language model and only have that appear once in the resulting graph
Basically what I want is that <unk> should never be preceded by anything in the history.
So suppose in get-text-counts we are getting counts for:
a b <unk> c d
and assuming 4-gram, then the sequence of n-grams consist of (in their natural order):
<s> -> a
<s> a -> b
<s> a b -> <unk>
<unk> -> c
<unk> c -> d
so <unk> should behave a little like <s> in how it appears in the history (i.e. nothing ever precedes it).
You will need kUnkSymbol.
This should be done as an option to get-text-counts. Currently the usage is
get-text-counts <ngram-order>
Now it should be:
get-text-counts [--limit-unk-history] <ngram-order>
This program reads lines of integerized text and outputs raw n-grams in
text form, one per line, in the format
<reversed-history> <predicted-word>
e.g.
6 5 7
See comments in code for more details, and get_counts.py for examples.
If the option --limit-unk-history is given, then any history greater
than bigram history that is to the left of <unk> (symbol number 3)
will be truncated (this relates to keeping decoding graphs compact
for Kaldi purposes).
There is a feature that I need for Kaldi reasons-- it will allow us to model
<unk>
with a phone language model and only have that appear once in the resulting graph Basically what I want is that<unk>
should never be preceded by anything in the history.So suppose in
get-text-counts
we are getting counts for:a b <unk> c d
and assuming 4-gram, then the sequence of n-grams consist of (in their natural order):so
<unk>
should behave a little like<s>
in how it appears in the history (i.e. nothing ever precedes it). You will needkUnkSymbol
. This should be done as an option to get-text-counts. Currently the usage isget-text-counts <ngram-order>
Now it should be: