stopwords filtering

[mikkokotila]

Have you thought of adding a stopwords/blacklist filtering option? I see there is already a whitelist, but I've found that generally in text analytics blacklist is used far more. It would be great to have it integrated into what already is a really wonderful (and well working!) solution.

[JasonKessler]

Thanks for the kind words.

There are a couple ways of removing stop words from your corpus, although function words can reveal interesting psychological traits. Moreover, modern lexicon mining techniques such as the Scaled F-Score and the Log-Odds-Ratio with a Dirichlet Prior can dampen the class-association score of highly frequent words.

If you'd still like to filter for stopwords, you can use the remove_terms function, as in term_doc_matrix.remove_terms(stop_words, ignore_absences=True).

Here's a complete, working example:

import scattertext as st
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
corpus = (st.CorpusFromPandas(st.SampleCorpora.ConventionData2012.get_data(), 
                              category_col='party', 
                              text_col='text',
                              nlp = st.whitespace_nlp_with_sentences)
          .build()
          .remove_terms(ENGLISH_STOP_WORDS, ignore_absences=True))

[mikkokotila]

Wonderful, thank you very much for this :) I will put it to test.