[Letters App] DataDog monitoring and alerts

[jacobworrell]

Value Prop

Monitoring product performance so we can address issues quickly.

Background

In order to maintain confidence in the Letters App as well as to predict and respond to issues, the error rate should stay below a low water mark. Above the low watermark is a warning sign. Above a high watermark is a problem. Error rates should be measured for each error type on each endpoint.

Notes:

• We don't have DataDog write access so we can't directly create monitors and alerts. 2 ways around this are:

  • Build out a temporary chart with the information needed to complete this work and work with Steve Albers to create the monitor; Steve has write access.
  • Build out a temporary chart and translate that into Terraform. Make a pull request to the devops repo

  Data Dog dashboard: https://vagov.ddog-gov.com/dashboard/86n-b39-hhn/benefits-management-tools-letters?refresh_mode=sliding&from_ts=1700515036918&to_ts=1701119836918&live=true

Acceptance Criteria:

• [ ] Low watermark identified w/ appropriate alerts sent to alert channel
• [ ] High watermark identified w/ appropriate alerts sent to alert channel
• [ ] Set up quick dashboard in DD and document query used
• [ ] Document DD config (Alert created and routed to #benefits-management-tools-notifications Slack channel, Chart created and added to Datadog dashboard, etc.)

[HeatherWidmont]

Engineers are meeting with @data-doge to discuss today

[HeatherWidmont]

Jerek & Sam walked Eugene through the products. Eugene will put together a rough proposed plan (possibly tickets), then review with the team & go from there.

[HeatherWidmont]

@samcoforma & @data-doge will be pairing on this after standup

[HeatherWidmont]

Closing, duplicate of https://github.com/department-of-veterans-affairs/va.gov-team/issues/73634