Philosophy Value ➡️ Problem Statement ➡️ Needs ➡️ Implementation Approach Trade-Offs ➡️ Inter-dependency.
In this issue, we want to organize a linear list of steps, serve as a landscape view of "how it is possible", to achieve what we want to achieve.
Philosophy, Value, Problem Statement & Ideas
End goal: we want to have a fb page that posts news from both US & China local media, to serve the purpose of voice equity and opinion diversity.
Mid term goal: to be more feasible, and also preserve the data, let's make sure we are able to obtain and keep the news data first, so we can do many things, we can simply just post in in fb like the end goal, or draw text cloud graph, or more advance NLP like Sentiment Analysis (:= whether data is positive, negative or neutral)
What data we want to keep: Everyday landing page, all stories linked by landing page, the text of a story. Image is not considered.
Caveat: landing page news might not change every day, some news seem to stick around for a couple days. It's likely that if we scrape everyday by landing page, we get many duplicated stories. ➡️ Let's still scrape landing page everyday, but we do de-duplication at story level; so one story only get scraped once.
Caveat: there're lots of casual, life-related columns that might seem deviating from our purpose, which is more about economy and politics. Ideally, you may categorize news first, then do word cloud or NLP within each category. But not sure how to effectively categorize, perhaps by section on the page, but this is highly dependent on the page structure, could be costly to maintain in the long run. ➡️ Not sure what's best way to deal with this. For now, just scrape it all.
Caveat: some news are more important than others, which may skew the word cloud. ➡️ our assumption is an important event should have more coverage and stories related so maybe not that bad. Still, we may also consider how many days a news stayed on landing page, from the assumption that important news usually sticks around longer.
Caveat: the site may have pay walls. ➡️ However, we determined the 1st paragraph could be still valuable. Or even just the title, will provide some insight.
Short term goals:
[x] Scrape & store everyday's landing page. Needs to do daily, most urgent.
[ ] Scrape all stories from landing page, do de-dup, so storing HTMLs of the stories. Secondary urgency due to risk of censorship. But otherwise can be done async.
Philosophy Value ➡️ Problem Statement ➡️ Needs ➡️ Implementation Approach Trade-Offs ➡️ Inter-dependency.
In this issue, we want to organize a linear list of steps, serve as a landscape view of "how it is possible", to achieve what we want to achieve.
Philosophy, Value, Problem Statement & Ideas
15