Spike: Blue/Green Deployment Improvements

[mmarcotte]

As a team, determine what improvements are needed for a blue/green deploy because presently there are many manual steps that require a great deal of precision and thought, and if some were automated they could help make the process easier and more reliable.

Pre-Conditions

• [x] Review this bug: https://github.com/ustaxcourt/ef-cms/issues/929 (DynamoDB records cannot be indexed into Elasticsearch until all of the data has made it to the destination table, otherwise we have incomplete information to call upon because some of it is still in transit.)

Acceptance Criteria

• [x] Perform usability study of the Blue/Green Migration in the Migration environment to walk through the steps, taking notes along the way. https://github.com/ustaxcourt/ef-cms/issues/922
• [ ] Review and improve the documentation in manual deployment https://github.com/ustaxcourt/ef-cms/blob/staging/docs/BLUE_GREEN_MIGRATION.md
• [x] Create stories for individual pieces of the Blue/Green Deployment process that need to change (be added or removed)

Notes

• Should also consider deployments that don't require a Migration. We ought not have to create a new ES cluster / DDB if we aren't going to use it.
• As we bring folks onto the team, it would be great to have a simpler, known, easy to learn deployment process.
  • Here are some ideas that could come out of our spike. https://github.com/ustaxcourt/ef-cms/issues/789

Ideas for usability study

• Create group of developers who commonly run deploys for both Flexion and the Court.
• For each developer:
• Run a complete deploy, including any local setup needed for manual account and environment-specific deployment steps, a migration, and color switching.
• Have another developer document every action, if action succeeds or fails on first try, and how long each step takes.
• Synthesize notes together to understand the most complicated, longest-running, and trickiest deployment steps to indicate where and how automation would be best applied.

[ericsorenson]

Consider using AWS Task Runners: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-remote-taskrunner-client.html

[codyseibert]

In a retro today we talked about maybe adding a prefix to the migration scripts, such as using batch1 batch2 bug-fix so we know more easily what these migration scripts are grouped together with.

[codyseibert]

We worked with Mike and discussed a couple of manual steps that could potentially be automated. The idea would be to group these up into different stories, with the following grouping potentially:

story 1:

• automate setting the migrate true flag in dynamo (maybe a new circleci workflow which sets the flag and deploys?)
• automate setting the destination table in dynamodb
• do not proceed if the segments dead letter queue has items in it
• automate read-only smoke tests & switch colors when ES is done (need to figure out a good way to know when ES is done reindexing without a 5 hour circle job running)
• make a "read only" smoke test for prod (we don't want smoketests creating / writing data, but we could at least try to look at cases, etc)
• keep track of which migration scripts have been ran
• automate destroying the migration infrastructure after everything was successful (only after switching colors)
• automate backing up and deleting the old dynamodb tables and ES cluster
• refactor terraform to not re-create the alpha / beta infrastructure if migrate = false (saves money)
• update (or consider deleting) the blue-green-migration.md since most of everything should be automated

story 2 (has user experience potentially):

• automate telling users the application is about to update (5 minute warning modal message)
• switch colors after the 5 minute warning
• force people to refresh their browser to continue to use the application