chucklesoclock
commented
9 years ago So haven't made much progress during spring break.
But what I will need for good-turing smoothing is a list of the awkwardly-worded "frequency of frequencies vector".
That is, if the text is "He uses statistics like a drunken man uses a lamp post—more for support than illumination," (Andrew Lang) then our feature vector fv will be:
- he --> 1
- uses --> 2
- statistics --> 1
- like --> 1
- a --> 2
- drunken --> 1
- man --> 1
- lamp --> 1
- post --> 1
- more --> 1
- for --> 1
- support --> 1
- than --> 1
- illumination --> 1
But the frequency of frequencies vector will be
- keys occurring once --> 12
- keys occurring twice --> 2
- keys occurring >= thrice --> 0
In general, if r = (a frequency value for keys), then I need an N_r = (how many keys occur r times).
Its main benefit is a output of the probability of unseen keys = N_1 / N, where N = (total number of keys).
There also might be code out there already that does some of the heavy lifting: http://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estimation#cite_ref-6
I'll get on it tomorrow
mtabor150
It's noted in all the journals as the best unigram smoother. Have skeleton of psuedo code written