Centralize Monitoring: Set up monitoring and alerting for platform services (in Datadog)

[jhouse-solvd]

Problem Statement

Monitors and alerts are spread across multiple systems. This negatively impacts the platform's ability to respond to incidents and support issues. And this leads to a confusing monitoring experience for platform operators. Additionally, maintaining multiple monitoring systems increases administrative overhead.

Background/context

• Prometheus is the Platform's legacy monitoring solution
  • connects to PagerDuty, Slack, and other communication channels
  • Deprecated in favor of Datadog
• There are still many monitors (rulesets) configured in Prometheus
• Prometheus alerts are configured using Alertmanager
• The Alertmanager configuration is here
• Datadog is the Platform's centralized monitoring solution
  • Note: Datadog is provided by VA OIT DevOps (DOTS)
  • Platform-facing Datadog documentation HERE

How might we...

• Centralize the platform's monitoring and alerting systems?
• Provide operators with tools to respond to incidents and support issues more effectively?
• Make alerts more actionable for support personnel?
• Improve the quality of life for on-call personnel?

Hypothesis or Bet

This initiative should...

• Provide a single source for metrics, monitoring, and alerting related to key platform services
• Simplify the Platform's monitoring and alerting system (and associated tooling)
• Prepare the Platform to deprecate Prometheus

We will know we're done when... ("Definition of Done")

• [x] Key platform components and services (listed below) have the following in Datadog:
  • [x] Dashboard
  • [x] Monitors
  • [x] Alerts
• [x] Key Platform components and services include
  • [x] AWS
  • [x] Network
  • [x] CI/CD
  • [x] Utilities
  • [x] EKS Infra
  • [ ] External services that the platform depends on
• [x] Operational documentation is updated (or created)
  • [x] to inform support and incident response personnel on where to find monitors and alerts
  • [ ] that identifies all components and services (in Confluence) (note: can use Mural board to populate)
• [ ] Monitors and alerts in Prometheus that are no longer needed have been identified and documented
• [x] There is a follow-up initiative to deprecate Prometheus: See #47083

Known Blockers/Dependencies

List any blockers or dependencies for this work to be completed

Projected Launch Date

December 31, 2022

Launch Checklist

Is this service / tool / feature...

... tested?

• [ ] Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
  • [ ] ... and issues discovered in usability testing have been addressed.
  • Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
• [ ] End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching
• [ ] (if applicable) New functionality has thorough, automated tests running in CI/CD

... documented?

• [ ] New documentation is written pursuant to our documentation style guide
• [ ] Product is included in the List of VSP Products
  • List the existing product that this initiative fits within, or add a new product to this list.
• [ ] Internal-facing: there's a Product Outline
• [ ] External-facing: a User Guide on Platform Website exists for this product/feature tool
• [ ] (if applicable) Post to #vsp-service-design for external communication about this change (e.g. customer-facing meetings)

... measurable

• [ ] (if applicable) This change has clearly-defined success metrics, with instrumentation of those analytics where possible, or a reason documented for skipping it.
  • For help, see: Analytics team
• [ ] This change has an accompanying Platform Release Plan.

When you're ready to launch...

• [ ] Configure your communication to teams about your work using the Change Communication Guide
• [ ] Conduct a go/no-go when you're almost ready to launch.

Required Artifacts

Documentation

• PRODUCT_NAME: directory name used for your product documentation
• Product Outline: link to Product Outline
• User Guide: link to User Guide

Testing

• Usability test: link to GitHub issue, or provide reason for skipping
• Manual QA: link to GitHub issue or documented results
• Automated tests: link to tests, or "N/A"

Measurement

• Success metrics: link to where success metrics are measured, or where they're defined (Product Outline is OK), or provide reason for skipping
• Release plan: link to Release Plan ticket

TODOs

• [x] Convert this issue to an epic
• [x] Add your team's label to this epic
• [x] Add this initiative to the Digital Experience Product Board using these instructions

[jhouse-solvd]

In speaking with @ph-One we should prioritize "top-tier" monitors to know which are the most important. Monitors, rulesets, alerts, metrics, etc should be listed in sequential order of importance. Then, issues can be created to tackle one by one.

[mchelen-gov]

Is there internal documentation that needs to be created or updated?

[jhouse-solvd]

Is there internal documentation that needs to be created or updated?

updated definition of done

[jhouse-solvd]

@ph-One and I were thinking that we may need to re-examine PagerDuty teams and services in light of the recent team restructuring.

[jhouse-solvd]

It might be better to scope this work to focus on Alertmanager rules related to critical monitors, ie "devops-critical" and "vsp-engineers-critical" (added to background context above)

[jhouse-solvd]

How this initiative is broken down:

• Epics exist for major platform components and services
• Epics contain individual issues for sub-components and services
• Issues are scoped to set up a dashboard, monitors, and alerts for the service/thing to be monitored

[jhouse-solvd]

@mchelen-gov - Can you add this initiative to the DE product board? It is currently in progress.