HPC | Notify team if GH/HPC sync fails

NDCLab / lab-devOps

NDCLab mgmt and operations

GNU Affero General Public License v3.0

0 stars 0 forks source link

HPC | Notify team if GH/HPC sync fails #223

Closed jessb0t closed 1 year ago

jessb0t commented 1 year ago

Need some mechanism to know when the nightly sync fails for a given repository.

jessb0t commented 1 year ago

Two-tier contact (initial and follow-up):

immediate notification of project lead (if one is specified) and the lab manager {for initial rollout}
after three days, escalates to notify lab manager

As project lead, you are being notified that there are changes on the HPC that have not been pushed to the GitHub remote. Please identify the source of these changes and complete the git sequence to re-sync with the GitHub remote.

jessb0t commented 1 year ago

If dataset with a failed sync does not exist in config-leads.json:

notify lab tech ("There is no one to notify about this sync failure.")
escalate to lab manager after three days

jessb0t commented 1 year ago

We added similar logic for any uncommitted/unpushed changes to the script.