2. Pre-processing text (For topic modeling)

[tmozgach]

• [x] Tokenize, normalize (lowercase)
• [x] Stop word removal (specific words for the domain)
• [x] Stemming: use weak morphological treatments? "At least for English, morphological conflation treatments such as stemmers and lemmatizers can worsen topic model quality on a variety of measures, while LDA turns out to be quite good at combining morpho- logical variants by itself."
• [ ] Bigrams ? https://markroxor.github.io/gensim/static/notebooks/lda_training_tips.html https://github.com/shichaoji/easyLDA/blob/master/build/lib/easyLDA/base.py (n_gram) https://www.datascience.com/resources/notebooks/data-science-summarize-hotel-reviews

[tmozgach]

Understanding Text Pre-Processing for Latent Dirichlet Allocation http://www.cs.cornell.edu/~xanda/winlp2017.pdf

[tmozgach]

Steps for effective text data cleaning (with case study using Python) https://www.analyticsvidhya.com/blog/2015/06/quick-guide-text-data-cleaning-python/