Closed githubssn closed 6 years ago
@githubssn awesome! just one small thing to add, do a punctuation removal to remove stopwords like
don't
at the moment it gets tokenized as 'don', '’', 't'
@heaven00 Thank you for your quick review.. I had a look again.. You are right. The issue you mention is in line 26. I believe I can delete that line. In the subsequent step, this issue is eliminated as the punctuation check is done subsequently - RegExpr Tokenizer is used in the line following which eliminates the punctuation issue and gives us distinct tokens.
:+1:
Looks good to me!
Thank you @githubssn and @heaven00 ! Merging.
POC for TF-IDF for initial English stories