Automate keeping data up to date by pulling data from upstream

nirabpudasaini commented 4 years ago

We want to automate collecting the data every day (or even every half-day?). Since upstream repo is update at 23:59 GMT (once a day), we can run our update script right after that time, eg, 00:00 GMT.

Acceptance criteria

[x] The repo is updated at least every day
[x] The new dataset is pushed to datahub.io/core/covid-19

Tasks

[x] Build action - https://github.com/datasets/covid-19/pull/15
- [x] Create github actions to:
- [x] setup python project
- [x] install dependencies
- [x] run the update scripts
- [x] commit changes and push to the repo (master branch)
- [x] Run it on a schedule at 00:00 GMT
- [x] Run it on master branch only
- [x] Setup github token so the action is authorized to push to the repo
[x] Deploy action (to datahub.io) - https://github.com/datasets/covid-19/commit/4c98133a49b5f26bfb0dc62a2e0ce8a31e36933c
- [x] prepare datapackage.json for the dataset
- [x] setup node project
- [x] install data-cli via npm/yarn
- [x] run data push command

Future

[ ] Run the update when upstream repo changes - seems non trivial to do if you don't control upstream see e.g. https://github.community/t5/GitHub-Actions/Triggering-by-other-repository/td-p/30668

larsonreever commented 4 years ago

this will help us a long way as we are coming up with a community based solution to track coronavirus near you - initiative named Corona Warriors, it will be an innovative step ahead to help the spread. Github has lots of good resources which we can levarage. Contributions & partnerships are welcome.

anuveyatsu commented 4 years ago

Mostly done - will close it once scheduled work succeeds. Note that we're now running the jobs every 6 hours instead of 24 hours because the upstream repo is not updated on regular basis.

anuveyatsu commented 4 years ago

FIXED see https://github.com/datasets/covid-19/actions/runs/61404499

zelima commented 4 years ago

Smth wrong, last 2 workflows are broken https://github.com/datasets/covid-19/actions

anuveyatsu commented 4 years ago

@zelima that was due to empty commits, eg, no changes in the upstream. I've now fixed it to succeed if here are no changes, see https://github.com/datasets/covid-19/commit/26567f9e71323968943815dce55326c9ea9be94c

datasets / covid-19