Read text processing book

[ariansajina]

Text Analysis in Python for Social Scientists

[ariansajina]

reading notes

• see introduction for useful references
• preprocessing
  • make all lowercase
  • clean
  • lemmatization
  • filter out stopwords
  • filter out based on POS (parts of speech)
  • join collocations
  • join words with their POS to disambiguate homonyms (e.g capital_NOUN vs capital_ADJ)
• TF-IDF counts
• topic models: LDA
• word embedding: word2vec
• pretrained embeddings: glove, fasttext
• document embedding: doc2vec
  • set document=speaker
  • visualize (dimensionality reduction)) and see if speakers cluster, if so, then annotate (institutions vs states, debtor vs creditor)
  • clustering (maybe k-means with k=2 and Yanis and Jeroen as starting centroids. but better agglomerative since small sample size)
• train markov n-grams on different speaker, then let their language models finish the same sentence

[ariansajina]

@book{hovy_2021, place={Cambridge}, series={Elements in Quantitative and Computational Methods for the Social Sciences}, title={Text Analysis in Python for Social Scientists: Discovery and Exploration}, DOI={10.1017/9781108873352}, publisher={Cambridge University Press}, author={Hovy, Dirk}, year={2021}, collection={Elements in Quantitative and Computational Methods for the Social Sciences}}