Create notebook for parsing and cleaning the mbox files - Githubissues

aicoe-aiops / fedora-mailing-list-analysis

This will be the repo for the Fedora mailing list sentiment analysis project

Other

0 stars 2 forks source link

Create notebook for parsing and cleaning the mbox files #3

Closed cdolfi closed 3 years ago

cdolfi commented 3 years ago

For each of the mbox files, the following needs to be completed:

[x] clean out the thread text
[x] removing the date portion prior to the thread text in body

need to figure out how to detect a date in the following formats to cut line:

"On Fri, 25 Jul 2003 22:55:53 -1000"
"name here wrote on Friday 08 August 2003 08:01:"
"On Mon, Aug 11, 2003 at 01:42:32PM -0700, name here"
"On Tue, 2003-09-23 at 10:41,"
[x] filter out known incorrectly formatted emails (no user text in them)
[x] strip "PGP SIGNED HASH" text from emails