Health Check Reliability

[TomElliottFlexion]

As a maintainer of DAWSON, I need the health check endpoint to return quickly enough not to cause timeouts on the status_health_check_west and status_health_check_east Route53 health check. As a maintainer of DAWSON, I need to be able to successfully re-route traffic between east and west regions when API Gateway or Lambda have gone offline in the region.

Pre-Conditions

https://app.zenhub.com/workspaces/flexionef-cms-5bbe4bed4b5806bc2bec65d3/issues/gh/flexion/ef-cms/10069

Acceptance Criteria

• The status_health_check_east and status_health_check_east Cloudwatch alarms associated with health checks of the same name to do not go into alarm state without a failure of AWS services.
• When the West API Gateway healthcheck route is unreachable or erroring, traffic will be rerouted to East API Gateway.
• When the East API Gateway healthcheck route is unreachable or erroring, traffic will be rerouted to West API Gateway.

Notes

Potential solutions

• Extend caching to service checks beyond SES.
• Share cache between endpoints (Lambda layer?)
• Define east/west urls in fqdn for health checks to hit instead of relying on latency-based routing
• Store health check value elsewhere (dynamo, s3, etc?) and check status after returning last value

Tasks

• [x] Investigate existing UI health checks for functionality/utility
• [x] Investigate if this aws_route53_record.public_api_route53_main_east_regional_record and aws_api_gateway_domain_name.public_api_custom_main_east are needed.
• [x] Revise Terraform implementation and see if it can be cleaned up
• [x] Verify stability of alarm/health check for at least a week (in test)
• [x] Fill in details for spike https://app.zenhub.com/workspaces/flexionef-cms-5bbe4bed4b5806bc2bec65d3/issues/gh/flexion/ef-cms/10121
• [x] Fill in all details of failover architecture diagram
• [x] Confirm that we can deploy with ENABLE_HEALTH_CHECKS on and off
• [x] Ensure health check does not fail immediately following deploy
• [x] Fix it: https://github.com/ustaxcourt/ef-cms/actions/runs/5884172351/job/15958372084?pr=3849 , https://github.com/ustaxcourt/ef-cms/actions/runs/5884172351/job/15958373004?pr=3849
• [x] Finish writing: https://app.zenhub.com/workspaces/flexionef-cms-5bbe4bed4b5806bc2bec65d3/issues/gh/flexion/ef-cms/10129

Test Cases

Story Definition of Ready (updated on 12/23/22)

The following criteria must be met in order for the user story to be picked up by the Flexion development team. The user story must:

• [ ] Is framed in business/user need, the value has been addressed.
• [ ] Includes acceptance criteria
• [ ] Has been refined
• [ ] Pre conditions have been satisfied. Process: Flexion developers and designers will test if the story meets acceptance criteria and test cases in Flexion dev and staging environments (“standard testing”). If additional acceptance criteria or testing scenarios are discovered while the story is in progress, a new story should be created, added to the backlog and prioritized by the product owner.

Definition of Done (Updated 5-19-22)

Product Owner

• [ ] Acceptance criteria have been met and validated on the Court's migration environment
• [ ] Add scenario to testing document, if applicable (https://docs.google.com/spreadsheets/d/1FUHKC_YrT-PosaWD5gRVmsDzI1HS_U-8CyMIb-qX9EA/edit?usp=sharing)

UX

• [ ] Business test scenarios have been refined to meet all acceptance criteria
• [ ] Usability has been validated
• [ ] Wiki has been updated (if applicable)
• [ ] Story has been tested on a mobile device (for external users only)

Engineering

• [x] Automated test scripts have been written, including visual tests for newly added PDFs
• [x] Field level and page level validation errors (front-end and server-side) integrated and functioning
• [x] Verify that language for docket record for internal users and external users is identical
• [x] New screens have been added to pa11y scripts
• [x] All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html
• [x] READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
• [x] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
• [x] Interactors should validate entities before calling persistence methods
• [x] Code refactored for clarity and to remove any known technical debt
• [x] Acceptance criteria for the story has been met
• [x] If there are special instructions in order to deploy into the next environment, add them as a comment in the story
• [x] Reviewed by UX on a deployed environment.
• [ ] Reviewed by PO on a deployed environment. Can be deployed to the Court's test environment if prod-like data is required. Otherwise deployed to any experimental environment.
• [ ] Deployed to the Court's staging environment.

[zachrog]

After a discussion between Mike, Jim, Tom, Zach, and Chris this story has been modified in service of a larger epic of making the DAWSON system more reliable/redundant. The broad strokes to achieving that epic are:

1. Create a DNS failover system that works for basic outages like Lamda and API Gateway.
2. Create a SPIKE to discover what infrastructure pieces we have in place for DAWSON, whether or not they are duplicated across regions, what it would take to make the infrastructure duplicated and capable of failing over to another region, the cost of duplicating resources, and the historic uptime of each system to assess its risk to the project.
3. Create redundant systems capable of accepting traffic from a failover, and create a monitoring system which is capable of redirecting that traffic to the healthy system.