Expand data generated in Training stage.

renepickhardt / generalized-language-modeling-toolkit

Generalized Language Modeling toolkit

http://glm.rene-pickhardt.de

52 stars 17 forks source link

Closed lschmelzeisen closed 9 years ago

lschmelzeisen commented 9 years ago

During training stage more data can and should be generated, so that it no longer has to be checked later:

NGramTimesCount
Distribution of sentence length
Things that get affected by only having a subset of all counts in cache (need to check those)
- NumWords
- VocabSize

lschmelzeisen commented 9 years ago

NGramTimesCount is counted by NGramTimesCounter.
Distribution of sequence length is calculated by LengthDistributionCalculator.
NumWords and VocabSize are counts of Patterns 0 and x. These just get copied from global counts if the are needed, but not manipulated.