Automate data update - Githubissues

danielabar commented 5 years ago

Currently have to manually download csv files from open canada. Enhance the process so it auto downloads, something like this:

Start at metadata https://open.canada.ca/data/en/dataset/dfa4daf1-669e-4514-82cd-982f27707ed0

Then follow each resources array entry, download url property.

paintedbicycle commented 4 years ago

This is a great idea. Not sure exactly how to do it - might be part of a build process for this repo? (i.e rather than an API call at runtime?). Would at least make it easier for you to republish each quarter and yet not rely on access to the Canada.ca site from the an app that consumes this package?

danielabar commented 4 years ago

I was thinking about a script that would automate all the steps currently outlined in the Maintenance section of readme. And then having a scripts entry in package.json so it could be run something like npm run data-update. So would still be some manual effort to kick it off, but a lot less than current.

Some complications could be:

Is it always the same url on the Open Government Canada site to download the latest csv's and metadata? If not maybe could be an input value to the script
What about test maintenance? Maybe need a way to isolate tests from constantly updating data. One possibility would be to introduce a data/test folder with some sample data that matches the production format, and then see if something like proxyquire could be used to replace the prod dependency const table1Single = require('./data/current/table1-single.json'); with the test folder when running tests. Thinking about this more, might warrant its own ticket, separate from the automation task.

danielabar / gis-lookup

Automate data update #4