Closed mikkokotila closed 6 years ago
Thanks for the kind words.
There are a couple ways of removing stop words from your corpus, although function words can reveal interesting psychological traits. Moreover, modern lexicon mining techniques such as the Scaled F-Score and the Log-Odds-Ratio with a Dirichlet Prior can dampen the class-association score of highly frequent words.
If you'd still like to filter for stopwords, you can use the remove_terms
function, as in term_doc_matrix.remove_terms(stop_words, ignore_absences=True)
.
Here's a complete, working example:
import scattertext as st
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
corpus = (st.CorpusFromPandas(st.SampleCorpora.ConventionData2012.get_data(),
category_col='party',
text_col='text',
nlp = st.whitespace_nlp_with_sentences)
.build()
.remove_terms(ENGLISH_STOP_WORDS, ignore_absences=True))
Wonderful, thank you very much for this :) I will put it to test.
Have you thought of adding a stopwords/blacklist filtering option? I see there is already a whitelist, but I've found that generally in text analytics blacklist is used far more. It would be great to have it integrated into what already is a really wonderful (and well working!) solution.