Infrastructure Protocols - Establish Game-day and Stress Test Protocols

[ricetj]

Description:

Establish gameday to test an aspect of or assertion against the platform. This will include learnings and remediation items as needed.

Pain Points being addressed:

• Clearer insight into the limits of our Platform.

User Story

As a VSP Ops team member, I want to be prepared for incidents that could occur, so that developer on VA.gov would have interruptions when helping veterans.

Goals

Test parts of our system to know how it reacts during an incident.

VSP OKRs

O3: Stability and resiliency of the Platform's systems and teams continue to improve.

KPIs

• Increase in customer NPS
• Reduction in volume of "non-human-needed" support requests month over month
• increase in the amount of data we have on our system

[pnwstevan]

AWS has recently open-sourced some internal chaos engineering tools which may be relevant/useful for this:

https://github.com/amzn/awsssmchaosrunner

https://aws.amazon.com/blogs/opensource/building-resilient-services-at-prime-video-with-chaos-engineering/

CC: @dginther @ricetj