Closed LindseySaari closed 4 months ago
Need to talk with support about available maintenance windows. Moving this ticket to the next Sprint.
Need to investigate how maintenance windows are set:
Terraform documentation: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticache_replication_group.html#maintenance_window
Thinking that we will have to also include DevOps. We can go Terraform, AWS consoles or automation for patches. @rjohnson2011 hopes to have this written up in the next day.
Here are discovery findings for PagerDuty automation for each of the following:
AWS can be integrated with PagerDuty via CloudWatch using the event rule "EC2 Instance System Maintenance" - https://support.pagerduty.com/docs/amazon-cloudwatch-integration-guide
A maintenance window argument is passed in terraform. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticache_replication_group.html#maintenance_window One possible way to automate this would be through Github Actions:
Below is example code for the following--
name: Update PagerDuty Maintenance Window
on:
push:
paths:
- 'terraform/**' # Trigger the workflow when Terraform files are updated
jobs:
update-maintenance-window:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Parse Terraform Files
id: parse-terraform
run: |
# Parse Terraform files to extract maintenance window settings
# Example: maintenance_window=$(parse_maintenance_window_from_tf_files)
echo "::set-output name=maintenance_window::$maintenance_window"
- name: Update PagerDuty Maintenance Window
uses: PagerDuty/pagerduty-action@v1
with:
token: ${{ secrets.PAGERDUTY_TOKEN }}
maintenance_window_id: ${{ secrets.PAGERDUTY_MAINTENANCE_WINDOW_ID }}
start_time: ${{ steps.parse-terraform.outputs.maintenance_window.start_time }}
end_time: ${{ steps.parse-terraform.outputs.maintenance_window.end_time }}
description: ${{ steps.parse-terraform.outputs.maintenance_window.description }}
This can be accomplished by sending PagerDuty a notification during automation for patches. The workflow for applying patch updates can send a PagerDuty API call during the initial stages of the process. When the patch update workflow is complete another PagerDuty API call can close the maintenance window and reactivate PagerDuty. Going to inquire during standup about more detail for the type of patch automation to occur to determine this workflow can support all use cases.
@LindseySaari and @Kshitiz-devops to review this next.
Nice work on this @rjohnson2011
My questions and comments:
It seems likely that we can integrate automated system maintenance events from RDS and ElastiCache and we should be clear to also include explicit details around these resource types. I think we should extend the details here to describe capabilities surrounding EC2 instances to other AWS resource types (RDS, Elasticache, other resources in our stack). I’m sure it’s relatively the same process as you detailed above. I really liked that you called out how to test out the PD <=> CloudWatch integration!
Regarding the automation of security vulnerability patches, can we apply the same approach you detailed for the PagerDuty + CloudWatch solution? It looks like automated patch events might also generate CloudWatch events, which we could potentially monitor for.
I know you had mentioned utilizing an event driven lambda or Github Action. Do you mind detailing that in the description here please?
Sync using terraform changes sounds good and we should rather be using terraform output to get the maintenance window rather than parsing the value I think. Lambda event based system for maintenance window sounds like good option. I would like to know more about it.
Is the Cloudwatch setup with pagerduty already tested? Will you need my help with that?
Here is some supplemental information addressing the additional comments:
The process to integrate AWS maintenance windows with PagerDuty for RDS and ElastiCache is similar to the one outlined for EC2 instances.
Set up the PagerDuty Integration in AWS as mentioned earlier. Create a CloudWatch Event Rule: In the CloudWatch console, go to "Events" > "Rules" and click on "Create rule." Under "Event Source," choose "AWS services" and select "RDS" as the service name. For "Event Type," select the "RDS Maintenance Window" event types. Configure the Target, Review, and Create the Rule following the same steps as for EC2.
Set up the PagerDuty Integration in AWS. Create a CloudWatch Event Rule: In the CloudWatch console, go to "Events" > "Rules" and click on "Create rule." Under "Event Source," choose "AWS services" and select "ElastiCache" as the service name. For "Event Type," select the "ElastiCache Maintenance Window" event types. Configure the Target, Review, and Create the Rule following the same steps as for EC2.
For security vulnerability patches, we can leverage the same approach of integrating CloudWatch events with PagerDuty. AWS services like EC2, RDS, and ElastiCache generate CloudWatch events when security patches are available or applied. We can create CloudWatch Event Rules to capture these events and trigger a PagerDuty incident or maintenance window. For example, for EC2 instances, we can create a CloudWatch Event Rule to capture the "EC2 Instance State-change Notification" event type and filter for the "system-reboot" event code, which is generated when an instance is rebooted after a security patch is applied.
In addition to the CloudWatch integration, we can also automate the process of creating and updating PagerDuty maintenance windows using event-driven architectures like AWS Lambda or GitHub Actions.
Create an AWS Lambda function that integrates with the PagerDuty API to create, update, or delete maintenance windows. Configure CloudWatch Event Rules to trigger the Lambda function based on specific events, such as when a maintenance window is scheduled in AWS or when a security patch is available. The Lambda function can parse the event data, extract the relevant details (e.g., maintenance window start and end times, resource IDs), and make the corresponding API calls to PagerDuty to manage the maintenance window.
Set up a GitHub Actions workflow in your infrastructure-as-code repository (e.g., Terraform) that triggers whenever Terraform files are updated or when a new maintenance window is defined. In the workflow, include steps to parse the Terraform files and extract the maintenance window settings. Use the official PagerDuty Action to update the maintenance window in PagerDuty based on the extracted settings from the Terraform files.
Both the Lambda and GitHub Actions approaches provide event-driven automation for managing PagerDuty maintenance windows, either based on AWS events or changes to your infrastructure-as-code repository.
Description
Our tools regularly undergo maintenance. Maintenance windows within AWS include various operations, including software updates, security patches, and system upgrades (planned and unplanned - example: a CVE patch). However, these maintenance activities are not currently reflected in PagerDuty, leading to potential alert fatigue and unnecessary escalations during planned downtimes. To streamline our incident management process and reduce noise for oncall folks, we need a solution to automatically sync AWS maintenance windows with PagerDuty maintenance windows.
Tasks
Success Criteria
Acceptability Criteria