Text analysis to-do - Githubissues

mark-williamson / LatNews-Scraper

0 stars 0 forks source link

Text analysis to-do #3

Open mark-williamson opened 6 years ago

mark-williamson commented 6 years ago

Most of this just makes sense to do in a separate file rather than at-source:

Remove line breaks and whitespace from text using NLTK (e.g. '\n', '\t')
Identify countries from either title or in-text
- named entity recognition for identifying countries: https://github.com/ushahidi/geograpy https://blog.ouseful.info/2017/09/04/simple-text-analysis-using-python-identifying-named-entities-tagging-fuzzy-string-matching-and-topic-modelling/
Filter out certain articles?
- e.g. based on titles like 'Tracking Trends', 'Leader', 'Quotes of the Week'
  - ... idk
Some stories begin with 'COUNTRY NAME |' and this could be removed before analysis

mark-williamson commented 6 years ago

Re: 3. ... there are some data-based articles that have no content title things like "Exchange rates" or "inflation in the region" that can probably by dropped

mark-williamson commented 6 years ago

Re: 1. ... This includes both body and title text. Some titles include whitespace markers as well.

mark-williamson commented 5 years ago

Take a look at very short stories and consider removing. Noticing in 2003 there are some brief snippets of one or two sentences that maybe do not warrant full article.

mark-williamson commented 5 years ago

5a. Some additional points from pre-2003 stories:

Note the '[No Title]' stories. My suggestion would be either remove these - they're single bullet points each, or add title as first X characters from story.
Same suggestion for the 'IN THIS WEEK'S REGIONAL REPORT' articles. Very short again.
Note a lot of titles are crammed together as 'word;NEW HEADLINE' and will need to consider separating at the semi-colon. The articles are normal otherwise.