Production Helm/AWS Secrets Manager Rollout

P0NDER0SA commented 2 days ago

Description

As a developer we need to safely deploy our migrated helm code to staging and production so that we can decommission our Kustomize code and gain the benefits of helm.

WHY are we building this? We are building this because it falls in line with industry standards and ties in with our OKRs for Secrets Management.

WHAT are we building? We are migrating all of our manifests code to Helm from Kustomize and reading secrets from AWS Secrets Manager rather than encrypted env files. We need to create scripts to do the migration safely and to minimize downtime.

VALUE created by this solution? The benefits to this are many including traceability of our deployment components (diff with helm) and more secure readable manifests code and variables/secrets.

Additional Information

The rollout plan looks like this:

Deploy all of the helmfile stuff with targetgroupbindings DISABLED - we will then have the kustomize and admin code deployed side by side, but no traffic will go to the new stuff. We can therefore verify that everything is up and running (kubectl port-forward to the pods).
Once confirmed everything looks good, we will run a small script that will delete the old target group bindings
We will enable the Helmfile target group bindings and deploy that
We will delete the rest of the Kustomize deployments
We should be able to minimize downtime to a 1-2 minutes this way.

Acceptance Criteria

Given some context, when (X) action occurs, then (Y) outcome is achieved

[ ] The rollout plan is scripted and reviewed by the team.
[ ] Verify alarms still work, so trigger a few manually.
[ ] Run a soak test in the target environment during the rollout so we can confirm and measure the downtime period.
[ ] Run performance tests to make sure the migration did not introduce any degradation.
[ ] Establish a release timeline to share with the GCNotify and support teams. Perform the rollout outside of business hours.
[ ] Communicate to users any downtime if more than 1h. Communicate with the product manager to sync on that.

QA Steps

[ ] Run rollercoaster tests in the target environment after the roll out.
[ ] Verify that the pods' configuration (especially the scaling metrics) match between Kustomize and Helmfile for the target environment.
[ ] Use the admin to run a few smoke tests such as sending an email and SMS.

P0NDER0SA commented 2 days ago

https://github.com/cds-snc/notification-attic/pull/71

P0NDER0SA commented 1 day ago

https://github.com/cds-snc/notification-manifests/pull/3144

jimleroyer commented 1 day ago

Pond wrote a 🍫 script in the attic. This is meant to perform the 2 of the 4 steps of the rollout to an environment.

cds-snc / notification-planning-core