search

OCHA-DAP / hdx-signals

HDX Signals
https://un-ocha-centre-for-humanitarian.gitbook.io/hdx-signals/
GNU General Public License v3.0
5 stars 0 forks source link

Create data dictionary #185

Closed caldwellst closed 2 weeks ago

caldwellst commented 3 weeks ago

Add script to create data dictionary that will be added to HDX. I have also added a workflow to update this, and did some slight updates to our other workflows to split out the update of static assets. We update all the locations data and metadata together in one, ending with the metadata file that is pushed to HDX. From there, the rest can be updated separately as necessary, including the new update_data_dictionary.yml flow.

hannahker commented 3 weeks ago

Overall changes look pretty good to me, but I'm not quite following the logic for why we're splitting the static data updates into separate files? Does this not add a bit of unneeded clutter in the scripts? We can still control the ordering of how they're updated from within the single file.

caldwellst commented 3 weeks ago

Yeah, previously was controlling the order, but actually, most of them are unrelated. The only ones that are critically dependent on each other are in update_locations_data.R, which is the ones that require first capturing how many locations we cover and then for each location, doing various calculations, finalising that with the overall locations metadata file.

The data dictionary file, ACLED info, and indicator mapping are not at all related, so figured do we need to do all the computation for the locations, like creating the ADM0 files, calculating centroids, etc. if we aren't updating them? Happy to put back all in one file but that was my thought process when I was creating the dictionary file because I was thinking, if we want to update the dictionary file, we would have to wait a long while for everything to update, even if there were no dependencies.