search

OCHA-DAP / hdx-signals

HDX Signals
https://un-ocha-centre-for-humanitarian.gitbook.io/hdx-signals/
GNU General Public License v3.0
5 stars 0 forks source link

Align CRON timings #116

Closed caldwellst closed 2 months ago

caldwellst commented 3 months ago

For HDX Signals, we are running CRON jobs on the IPC API, similar to the workflow here. Godfrey flagged that we had previously sent a signal for the IPC that was not reflected on HDX until the next day because we have not aligned the timing of our CRON jobs.

Our runs are currently on weekdays at 11 UTC, while the HDX scraper is every day at 8:10 UTC.

Given that we want to have time to review alerts with Sarah, I thought might be best to start our runs at 8UTC, which will be just before the workday begins for most of us in UK/CET time and give sufficient review time to send alerts before working day is out in most areas we work, and for the morning time in the USA. This would also ensure that we also run prior to the IPC scraper so no issues of alignment will occur there. For other datasets:

  1. JRC ASAP, we are checking HDX data, which is updated at midnight UTC so will be aligned there.
  2. IDMC IDU is run at 19:45 UTC, so there is a risk of misalignment. Will raise issue in rep to check about this.
  3. ACLED, the scraper repo I could find is archived, so will ask Mike what the current setup is there.
  4. WFP market monitor, we will be using HDX, so no misalignment there.
caldwellst commented 2 months ago

Mike aligned his IDMC CRON job with us: https://github.com/OCHA-DAP/hdx-scraper-idmc-idu/issues/1. Merging now.