Discovery: Escalate Datadog Configuration Drift to Devops Tier 2 Support

[flooose]

User Story

As developers on the Platform Product team We need to reconfigure the active alerts in Slack to a Tier 2 support issue So that Tier 2 Support is aware of DataDog configuration drifts that are happening in real time, and those drifts can be remedied in a timely manner.

Summary

We currently have a custom datadog drift checker that's easy to ignore/lose track of in the devops-alerts Slack channel. In order to keep configuration drift of our datadog monitors in check, we want to post a Slack message to the DSVA Support channel, ideally escalated directly to tier 2.

Tasks

• [ ] Coordinate this with the support channel. It's still unclear if we can programmatically create an issue in this channel.
• [ ] Address the possibility of escalating to devops tier 2 directly. Since we rotate a different person into tier 2 support, we need some mechanism of alerting them. Alternatively, we can word the message in such a way that informs tier 1 that they may immediately escalate to tier two.
• [ ] Possible solution: the Tier 2 engineer as in this Slack example
• [ ] What are automated solutions for this issue?
• [ ] Tie in solution with PagerDuty (API call to get the current support engineer)

Success Metrics

• [ ] Confirmed whether or not this work can be done in the Support Channel
• [ ] Process has been tested by the new resources responsible for escalating configuration drift
• [ ] Documentation is drafted by resources and PM and approved by those responsible for performing the work.

Acceptability Criteria

• [ ] Established a new process for managing Datadog Configuration Drift when it has been noticed in code.

• [ ] New process for managing DataDog Configuration Drift has been documented, and a RACI or Roles & Responsibilities document has been drafted and approved. @jennb33 is happy to help with this RACI/R&R documentation.

Validation

Assignee to add steps to this section. List the actions that need to be taken to confirm this issue is complete. Include any necessary links or context. State the expected outcome.

[jennb33]

@flooose has made a connection to PagerDuty, then will work on posting to Slack, then will coordinate with the support team on how to implement.

[jennb33]

6/27 Stand-up update: @flooose should be able to start testing the Slack connection, then we need to determine where the message needs to go. Might send it to devops-cop for a pre-review.