Discovery: Investigate 4xx Errors that cause the prod.cms.va.gov EC2 Instance to Refresh After Hours

[7hunderbird]

User Story or Problem Statement

Our team needs to investigate why the load balancer that sits above the prod.cms.va.gov EC2 instance has alerted.

Moreover, that it appears that upon initial investigation the prod.cms.va.gov EC2 instance had automatically replaced itself (through an Auto Scaling Group lifecycle event) at the time that the load balancer alerts happened.

Description or Additional Context

On Aug 7th @ph-One made our team aware of a monitoring alert that happened after 8PM ET on Aug 6th.

The "Platform - AWS Application Load Balancer % of HTTP 4xx Responses" alert tracks if there has been a higher than 25% amount of 4xx responses downstream from the load balancer. This means that anything the load balancer is trying to reach it's failed to make a connection to those resources.

400 series error code can have a variety of issues including:

• 408 Request timeout
• 403 Forbidden
• or perhaps the most famous of all... 404 file not found

In any case, if a Load Balancer can't find what it expects to find behind it, then it's important to know about.

Steps for Implementation

Acceptance Criteria

• [ ] Determine what is triggering the lifecycle event to replace the prod VM
• [ ] Make recommendations on next steps or action items depending on what the trigger is
• [ ] If the trigger is an external system (i.e. Jenkins), perhaps we reconfigure that system
• [ ] If the trigger is an internal process we examine what that process is and options to reconfigure

[7hunderbird]

• link to slack

• source

[gracekretschmer-metrostar]

Next steps: create a BRD epic for CMS infrastructure research and small tasks for improvements. Pull in Tyler's tasks into that epic.