Centralize Monitoring: Provide write access for VFS backend engineers and deprecate legacy monitoring

[jhouse-solvd]

Problem Statement

VFS Teams need to be able to monitor their applications in Datadog.

However, currently:

• Metrics, monitors, and dashboards are spread across multiple systems (Prometheus, Grafana, Sentry, and Datadog), making information difficult to correlate by VFS teams
• This leads to decreased visibility into the performance and behavior of VFS applications, especially from the perspective of the VFS backend engineers (and thus, their product teams) developing these applications
• The Platform has chosen Datadog as the 'single-pane-of-glass' monitoring tool but has not yet granted write permissions to VFS teams
  • Without properly scoped permissions, VFS teams could inadvertently re-configure others' dashboards and monitors, including those needed by Platform teams for ongoing observability and monitoring
  • Datadog lacks built-in features that allow the Platform to scope permissions (ie write access) to app- or team-specific resources, which means a solution is needed to enforce Datadog configuration standards and avoid disruption to resources in Datadog (i.e., dashboards, monitors, alerts) that the Platform depends on for day-to-day operations
• Additionally, multiple legacy monitoring systems have to be maintained by Platform, creating maintenance and administrative overhead and increasing technical debt
  • VFS backend engineers need parity in Datadog before legacy monitoring tools can be decommissioned

How might we...

• Consolidate application performance and behavior monitoring for VFS teams?
• Provide write access to VFS teams without disrupting Platform observability and monitoring?
• Reduce the number and variety of systems that must be maintained?
• Remove dependencies on legacy systems so that those systems can be deprecated and sunset?

User Impact

• VFS engineers are blocked from accurately understanding the performance and behavior of VFS applications
• VFS engineers are blocked from developing and enhancing monitors and alerts for VFS applications

Where was this problem reported?

• Reported frequently in Postmortems and support issues from both Platform and VFS teams.
• Captured during UX discovery research performed by Platform Tech Team 2 in late 2022 and early 2023.
• Frequently requested by VFS teams, especially in the first quarters of 2023.
• One Slack discussion that is pertinent to this.

How well do we understand the problem?

We understand the technical problem space well and have identified potential solutions. Areas that need further investigation:

• What roles and permissions do VFS teams need to leverage the monitoring system effectively?

Considerations

• Grafana is used for both monitoring (i.e., dashboards) and logs
• However, only monitoring has been addressed within the scope of this project
• That means that Grafana should be deprecated for monitoring only until all logs are moved to Datadog as (i.e., as part of a separate project/initiative)

Acceptance Criteria (AC)

• [ ] A single centralized monitoring tool provides the following to VFS Backend engineers:
  • [ ] Metrics collection and aggregation
  • [ ] Dashboards for VFS application performance and behavior
  • [ ] Monitoring of VFS applications
  • [ ] Notifications (i.e., alerts) to various communication channels
• [ ] The monitoring tool is accessible to VFS Backend engineers, along with supporting processes and documentation
  • [ ] Documentation (i.e., instructions and guidance) for how VFS teams can use the tool
  • [ ] Onboarding and offboarding processes exist and are documented
  • [ ] VFS Backend engineers can use the tool to create dashboards and monitors for their applications without impacting others
• [ ] Legacy monitoring resources have been deprecated:
  • [ ] Relevant dashboards are deprecated in legacy systems (i.e., removed from Grafana if no longer required)
  • [ ] Relevant metrics/monitors/rulesets are deprecated in legacy systems (i.e., removed from Prometheus if no longer required)
  • [ ] Grafana Documentation (specifically, for monitoring, not logging) is removed for VFS teams (i.e., from the Platform Website)
  • [ ] Timeline for the sunset of Grafana is created

How should we measure success?

• How many VFS backend engineers have access to create dashboards and monitors in Datadog?
• How many dashboards have been created by VFS Backend engineers?
• How many monitors have been created by VFS Backend engineers?
• How many dashboards have been deprecated (or removed) from Grafana?
• How many rules have been deprecated (or removed) from Prometheus?

TODOs

• [x] Convert this issue to an epic
• [] Assign project team name
• [ ] Identify this project's OCTO and Vendor key POCs
• [ ] Identify specific VFS teams to be part of this project's acceptance
• [ ] Identify project target end date
• [ ] Conduct project kickoff meeting
• [ ] Develop and implement a project communication plan

[batemapf]

re:

The Platform has chosen Datadog as the 'single-pane-of-glass' monitoring tool but has not yet granted write permissions to VFS teams Without properly scoped permissions, VFS teams could inadvertently re-configure others' dashboards and monitors, including those needed by Platform teams for ongoing observability and monitoring Datadog lacks built-in features that allow the Platform to scope permissions (ie write access) to app- or team-specific resources, which means a solution is needed to enforce Datadog configuration standards and avoid disruption to resources in Datadog (i.e., dashboards, monitors, alerts) that the Platform depends on for day-to-day operations

it looks like you can restrict editing of monitors to specific roles (though i assume those roles are defined by the DOTS team, so maybe not practical). alternatively / additionally, alerts can be configured to trigger when a monitor is edited. might not be a perfect solution:

[mchelen-gov]

re:

The Platform has chosen Datadog as the 'single-pane-of-glass' monitoring tool but has not yet granted write permissions to VFS teams Without properly scoped permissions, VFS teams could inadvertently re-configure others' dashboards and monitors, including those needed by Platform teams for ongoing observability and monitoring Datadog lacks built-in features that allow the Platform to scope permissions (ie write access) to app- or team-specific resources, which means a solution is needed to enforce Datadog configuration standards and avoid disruption to resources in Datadog (i.e., dashboards, monitors, alerts) that the Platform depends on for day-to-day operations

it looks like you can restrict editing of monitors to specific roles (though i assume those roles are defined by the DOTS team, so maybe not practical). alternatively / additionally, alerts can be configured to trigger when a monitor is edited. might not be a perfect solution:

Yup, we're trying to figure out how to get the roles configured and if they can be mapped from AD or other SSO.

[little-oddball]

Would encourage the addition of at least 1 spike related to role decomposition and experimentation.