Set up 1 alert with Pager Duty & Slack

[QaysarA]

Value Statement

As a developer I want to receive alerts on app metrics via PagerDuty and Slack So that I can respond appropriately

(This card is the first of setting up alerts and will set up one initial alert with the two alerting systems. After this, #152 covers the other metrics we want to alert on.)

Acceptance Criteria

GIVEN an alert notifies the team about a concern WHEN the developer receives the notification via PagerDuty THEN they are informed enough to act on the notification.

GIVEN an alert notifies the team about a concern THEN an alert is sent via Slack.

Checklist:

• [x] DataDog Set Up
  • [x] Create a dashboard of with the initial metric, using terraform
  • [x] Configure initial alert: Measure whether the heath check is returning success or failure. On failure it should give an alert on both PagerDuty and Slack
• [x] PagerDuty
  • [x] Get access from VA - contact DOTS team
  • [x] Set up team and escalations of who should be notified (not setting up on-call schedule at this time)
  • [x] All team members add their notification preference (email/text)
    • [x] @philherbert
    • [x] @ffafara-tw
    • [x] @marisahoenig
    • [x] @saman-moshafi-va
    • [x] @lingtran
  • [x] Alerting is integrated with PagerDuty
• [x] Slack - to private #notifications channel
  • [x] DataDog Slack Integration

Assumptions

• ~For now, we will set up alerts with CloudWatch and move to DataDog later.~ Since we want to move to DataDog eventually and have the alerts all in one place, we should set them up in DataDog now instead of putting work into CloudWatch first and migrating.

Additional Info/Resources

• Research from the spike #151 in Google doc, explaining the reasoning and how to set up each alert.
• Helpful PagerDuty setup video
• DataDog monitor terraform
• Refer to ADR 6 in vanotify-team repo

Out of Scope

• Creating an on-call procedure in PagerDuty - bigger conversation needs to occur

Open Questions

• Do we want to see metrics/get alerts for all environments?

[philherbert]

Blocked while we're waiting on getting access to pagerduty

[miabecker]

Seems like there is a lag in datadog now so need to consider cloud watch @miabecker