Datadog alert integration with SNOW proof of concept

[mchelen-gov]

Problem Statement

VA.gov and monitors and alerts are configured in Datadog, however all VA incident response is managed through Service Now (SNOW). We want to understand what integrations exist or are available to send data between these systems.

Impact

• When upstream systems are encountering problems, having this information pulled into Datadog would speed up identification and resolution of issues.
• VA.gov is seen as a collaborative participant in enterprise information sharing and incident management
• VA.gov benefits from MIM process (e.g. by escalating issue with upstream systems) since ECO and the MIM process is driven by SNOW.

Success Criteria

• ECC is able to respond to alerts about VA.gov
• Coordination is provided between VA.gov teams and MIM process
• Significant VA.gov status information is communicated to ECC

Involved

ECC: Jeff Sleeper, Luke Bader, Swami (Prasad) Gorle Butchi OCTO: @mchelen-gov @BillChapmanUSDS @va-albers @AparnaNittalaUSDS

Relations

• Follow up to #67591

[mchelen-gov]

Proof of concept

When a monitor is triggered in Datadog, it sends an "alert" to Service Now using a webhook. The Webhook-API-yourit-va-gov webhook has been installed and configured with necessary auth.

Based on discussion with ECC/BLM, our approach was to pick a single monitor and get it fully configured with SNOW webhook, tags, and necessary information for ECC Event Management team to be able to respond to alerts.

The monitor that was selected:

• Monitor: VA.GOV – VASI_2103 – PROD – VA.GOV HomePage – Synthetic Health Check
• Synthetic test: VA.gov Homepage Synthetic - Production

Integration with SNOW is confirmed to be working, and alerts generated from Datadog can be viewed in SNOW:

• SNOW - View of all VA.gov alerts in SNOW

• Datadog - Example alert from Jan 17 Event ID: 7403633220778450139

• SNOW - Sample triggered alert in SNOW: Alert3407211

Placeholder instructions for ECC:

** ECC Event Management **
Monitor is triggered when https://www.va.gov fails to load, or is missing required elements.
Screenshot of test failures can be seen here: https://vagov.ddog-gov.com/synthetics/details/u4h-eqw-qmn?monitorStatus=fail
Please report to: vagovdevops@va.gov

See Also

• All Datadog monitors configured with webhook

[mchelen-gov]

This completes the implementation of the proof of concept. The established monitor can now be modified as needed to suite the needs of OCTO, Platform, and ECC.

Recommended Next Steps

• OCTO (DE/Platform): Update alert text to reflect desired actions by ECC Event Management team (Slack).
• ECC (Prasad): Update webhook configuration to ensure Severity field is mapped correctly from Datadog.
• ECC (Luke / Jeff): Request installation of full Datadog SNOW integration (more robust and configurable than webhook) - DMND0005947