search

christian-westbrook / interactive-sentence-predictor

An interactive sentence predictor that generates likely continuations of partial sentences.
0 stars 0 forks source link

Improved tokenizer #7

Open rfishe02 opened 6 years ago

rfishe02 commented 6 years ago

The tokenzier is only suited for the previous Problem Set that was assigned to our class. I may have a look at it this weekend to see whether I can make some improvements.

rfishe02 commented 6 years ago

I made some slight improvements to the the tokenizer. Now, we're ignoring tags & special html characters, such as &...;.