Refactor to ignore a failure for the grace period

GOCDB / gocdb-failover-scripts

Scripts and doc for hosting a failover

1 stars 2 forks source link

Refactor to ignore a failure for the grace period #18

Closed gregcorbett closed 1 year ago

gregcorbett commented 2 years ago

currently, an error in the pipeline immediately causes the next icinga check to raise an error.
this led to flapping hour to hour, generating emails that were ignored.
this should mean we only get alerts if the failover pipeline has not succeeded within the grace period.
adding an output on a successful restore in 1_runDbUpdate.sh simplifies the logic needed by this check. We don't have to worry about "if/when the pipeline failed?", we only care about "when did it last succeed?".

gregcorbett commented 2 years ago

Subject to a positive review, I'll open an issue that the MariaDB version of the pipeline should also output a timestamp on a successful completion (didn't wan't to do it in this PR because I'm not in a position to test any changes to that side of things)

gregcorbett commented 1 year ago

A unit test would be nice if possible.

See #21 😄 . I'll close this PR in favour of that one.