cds-snc / notification-planning-core

Project planning for GC Notify Core Team
0 stars 0 forks source link

Improve DB alarm to indicate that the DB is down #68

Open jzbahrai opened 1 year ago

jzbahrai commented 1 year ago

Description

As a user, I want to know if the DB is down. Our alarms should go off if the DB is down, not just when there are celery errors. This is a followup from this incident: 2023-02-21-celery-errors-database-upgraded-automatically, where the devs discovered the DB was down because of celery errors (and wouldn't have otherwise).

Acceptance Criteria

QA Steps

ben851 commented 7 months ago

I've investigated this a bit and it seems as thought we're already subscribed to aws rds events for failures etc.

I'm going to investigate whether there are some secondary metrics we can use to infer an issue.