Automate data source collection and aggregation

bobbywlindsey commented 4 years ago

Setup scripts to gather data from generally sources like:

Johns Hopkins data sources ✅
"Detailed data" Google Sheet ✅
https://github.com/beoutbreakprepared/nCoV2019
https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports (Chen is doing this manually right now to produce the data in 2019-nCov Stat International)
https://www.worldometers.info/coronavirus/ (this gives us an overview but JH data is better for time series stuff probably)
Netherlands data can be downloaded from here: https://www.volksgezondheidenzorg.info/onderwerp/infectieziekten/regionaal-internationaal/coronavirus-covid-19#node-coronavirus-covid-19-meldingen

One will need to update the database/covid19_data_aggregator library to scrape the data as well as the database/run_update.py script to implement the new functionality in daily updates.

olynch commented 4 years ago

We should get this data into the ElasticSearch server as well.

bobbywlindsey commented 4 years ago

@olynch Yeah we can definitely do this once we start aggregating data from various sources

olynch commented 4 years ago

This should not have been closed by that PR, there are still things to do regarding this.

BWStearns commented 4 years ago

I'll take worldometers.

ahl54 commented 4 years ago

I got https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports

necsi / covid19

Automate data source collection and aggregation #1