Language Model Interface

[tejasvaidhyadev]

language model inspired by NLTK.lm package

Roadmap

• [X] vocabulary - Implemented contain vocabulary structure

• [x] Counter - contains language model counter

• Structure and methods for counting ngrams and will count any ngram sequence you give it

• [x] models - contains Interpolated Language model

• KneserNeyInterpolated :heavy_check_mark:

• Laplace :heavy_check_mark:

• Lidstone :heavy_check_mark:

• MLE :heavy_check_mark:

• WittenBellInterpolated :heavy_check_mark:

• [x] preprocessing - contains all function needed for preprocessing

• [x] smoothing - Smoothing algorithms for language modeling.

• [x] utility

• [x] documentation and test

it will also include

• Cross entropy of model for given eval text
• logscore - eval the log score of word in the context
• Perplexity - perplexity of the given text

[tejasvaidhyadev]

Now we have working MLE or Ngrambase model

[Ayushk4]

Once you are finished with the PR, try to incorporate above suggestions. Let me know once you are done.

[tejasvaidhyadev]

Thanks, @Ayushk4 I will incorporate the changes asap.

[Ayushk4]

Hi, Tejas. Thanks for incorporating the changes. Please let me know once this PR is ready for review.

[tejasvaidhyadev]

Hi @Ayushk4 you can review PR now Thank you

[Ayushk4]

Travis seems to fail on 1.3: ERROR: Ambiguous dependency on StatsBase. Can you look at this once?

[tejasvaidhyadev]

Travis seems to fail on 1.3: ERROR: Ambiguous dependency on StatsBase. Can you look at this once?

updated

[Ayushk4]

@aviks Can you have a look and suggestion changes? Maybe then we can have this merged?

[tejasvaidhyadev]

@aviks can we merged it?

[aviks]

Taking a look now