2.1 - A and B

[sonali-sr]

2.1 Preprocess the data by removing stopwords, punctuation, and non-alpha words (5 points)

A. Write a function that:

• Takes in a single raw string in the contents column from that dataframe
• Does the following preprocessing steps:
  • Converts the words to lowercase
  • Removes stopwords, adding the custom stopwords in the code cell below to the default stopwords list
  • Only retains alpha words (so removes digits and punctuation)
  • Only retains words 4 characters or longer
  • Uses the snowball stemmer from nltk to stem
• Returns a joined preprocessed string (so if press release is something like "The CEO was indicted" it might return "ceo indict"

B. Use apply or list comprehension to execute that function and create a new column in the data called processed_text.

Note: there will be a deduction if your code uses a non-list comprehension for loop that uses append.

Resources:

• Here's code examples for the snowball stemmer: https://www.geeksforgeeks.org/snowball-stemmer-nlp/

• Here's code with topic modeling steps: https://github.com/rebeccajohnson88/PPOL564_slides_activities/blob/main/activities/fall_22/solutions/09_textasdata_partII_topicmodeling_solution.ipynb

[sanhatahir]

Just a quick question; With your given example of "The CEO was indicted", wouldn't CEO be removed because of the 4th condition of words with length >= 4?

[rebeccajohnson88]

Just a quick question; With your given example of "The CEO was indicted", wouldn't CEO be removed because of the 4th condition of words with length >= 4?

yes good flag! that was just an example :)