Create New Datadog Synthetic Monitors for Production Services - Githubissues

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

284 stars 206 forks source link

Create New Datadog Synthetic Monitors for Production Services #46016

Open pjhill opened 2 years ago

pjhill commented 2 years ago

Issue Description

Create additional synthetic monitors in order to monitor and alert on production service outages.

Consider proactively detecting issues like this: https://dsva.slack.com/archives/C30LCU8S3/p1660402315487859

Known blockers:

Production test accounts are not available
VA generally dislikes traffic to production services for testing purposes

Tasks

[ ] Evaluate and list high impact external service integrations
[ ] Address issues of obtaining accounts to use for the production synthetic monitors with management
[ ] Design synthetic monitors for selected high impact production service integrations

Acceptance Criteria

[ ] Production test account issues has been resolved
[ ] High impact production service integrations are monitored by Datadog synthetic monitors

JoeTice commented 2 years ago

Updating to high priority based on Chelen's request to prioritize this