OCHA-DAP / hdx-scraper-idmc-idu

MIT License
0 stars 0 forks source link

CRON alignment with HDX Signals #1

Closed caldwellst closed 4 months ago

caldwellst commented 4 months ago

For HDX Signals, we are checking to ensure that we have aligned CRON jobs across the datasets we use. We have aligned our food security updates based on something Godfrey noticed, see this PR.

However, I wanted to check about the logic behind the 19:45 UTC scheduling for this scraper to see if there is a risk of misalignment between HDX Signals and HDX data if we run at 08:00 UTC. Obviously there might be misalignment if we expect updates to the IDMC IDU between those times, but seems relatively unlikely given the IDU is managed out of Geneva.

mcarans commented 4 months ago

@caldwellst The timing I have set is arbitrary and I can change it to whatever works for Signals. Please let me know what would work best.

caldwellst commented 4 months ago

Thanks @mcarans. Could we move to 08:10 UTC like done here please? That way we can be confident we will have downloaded and processed our data prior to your run.

mcarans commented 4 months ago

Done. Please close if all good.