Sync AWS Maintenance Windows with PagerDuty

jennb33 commented 7 months ago

Description

Our tools regularly undergo maintenance. Maintenance windows within AWS include various operations, including software updates, security patches, and system upgrades (planned and unplanned - example: a CVE patch). However, these maintenance activities are not currently reflected in PagerDuty, leading to potential alert fatigue and unnecessary escalations during planned downtimes. To streamline our incident management process and reduce noise for oncall folks, we need a solution to automatically sync AWS maintenance windows with PagerDuty maintenance windows.

This ticket is intended to capture maintenance window events and plan for automation for PagerDuty.

Tasks

[ ] Research integration options - How do we integrate this with Pagerduty
[ ] Develop the necessary configurations required to support the automatic syncing of maintenance windows
[ ] Document the integration process. Ensure that the team is aware of how to schedule maintenance windows in AWS with the new syncing process in place.

Success Criteria

[ ] That there is a path forward with synching AWS Maintenance Windows with PagerDuty, or that it is determined that it is not possible to do so.

Acceptability Criteria

[ ] A maintenance window scheduled in AWS automatically creates a corresponding maintenance window in PagerDuty
[ ] Updates or cancellations to AWS maintenance windows are reflected in PagerDuty
[ ] Documentation is provided and up to date

rjohnson2011 commented 5 months ago

Picking up today

jennb33 commented 5 months ago

This is the command that @rjohnson2011 is running: aws ssm describe-maintenance-windows --output json > maintenance_windows.json

jennb33 commented 5 months ago

@rjohnson2011 is working with @LindseySaari and @Kshitiz-devops This may need a credential for PagerDuty in order to link it to PagerDuty, @rmtolmach may be an admin that can help; @Kshitiz-devops is an admin Look at GitHub Action Secrets? Kshitiz did that Kshitiz will create a new credential so that Ryan can make a post request and update the maintenance windows.

jennb33 commented 5 months ago

Duplicated by 85083 for 1 story point in Sprint 2

department-of-veterans-affairs / va.gov-team