OCHA-DAP / hdx-signals

HDX Signals
https://un-ocha-centre-for-humanitarian.gitbook.io/hdx-signals/
GNU General Public License v3.0
5 stars 0 forks source link

Fix static data updates #92

Closed hannahker closed 3 months ago

hannahker commented 3 months ago

Minor updates to ensure that the static data update runs from the GitHub action with informative logging and status updates.

Issues still to resolve:

If we're assuming that all files under inputs/ in the storage container should be updated by this workflow:

caldwellst commented 3 months ago

The action should fail when R errors out

This is an issue with using find in GitHub Actions. This is why in the original file we were using globstar. See this https://apple.stackexchange.com/questions/49042/how-do-i-make-find-fail-if-exec-fails

caldwellst commented 3 months ago

Connection errors when running action (for example)

Does this work when running locally?

caldwellst commented 3 months ago

un_geodata.geojson: See #88

idmc_country_links.parquet and indicator_mapping.parquet. I put those together manually, just checking all country links on the IDMC website, and then piecing together indicator_mapping.parquet as I was developing. I think we can hardcode an update script that we adjust as necessary. A really nice and easy trick is that you can generate the code to create the data frame in R super easily. Just load it in, save to df and then use dput(df). However, it's not that clean / easy to update, whereas dplyr::tribble() is a nice way to rowwise create data. I think we should do it for sure.

country_taxonomy_backup.parquet Remove!

hannahker commented 3 months ago

@caldwellst

Aha nice find!

This is an issue with using find in GitHub Actions. This is why in the original file we were using globstar. See this https://apple.stackexchange.com/questions/49042/how-do-i-make-find-fail-if-exec-fails

It does! I'm thinking that this could be from the {} in the DSCI_AZ_ENDPOINT environment variable. I'm thinking we can just hard-code this in the code itself since this doesn't contain any sensitive data.

Does this work when running locally?

hannahker commented 3 months ago

@caldwellst this should be ready for final review now. As discussed, we'll come back to the ACLED data issues in future work. For now, I've left the bash command as using find, so that this script can run through all other update files, despite the ACLED data failing. See #104