ben851 commented 1 year ago

Description

As a developer/operator of GC Notify, I would like my infrastructure to be up to date with EKS versions so that we are never racing against the clock to upgrade our systems before their versions are deprecated.

WHY are we building?

Being several versions ahead of the minimum version will allow us to stay ahead of security patching, features, and prevent situations where we need to upgrade our cluster extremely quickly.

WHAT are we building?

Upgrade the scratch, staging, and production K8s clusters to

[x] 1.24
[x] 1.25
[x] 1.26
[x] 1.27

VALUE created by our solution

More secure, better reliability, the elimination of tech debt.

Acceptance Criteria

[x] Scratch K8s is upgraded to 1.27 and notify still works
[x] Staging K8s is upgraded to 1.27 and notify still works
[x] Prod K8s is upgraded to 1.27 and notify still works

QA Steps

[x] Run smoke/soak test against each environment
[x] verify that the admin / api k8s / dd-api / celery logs are all working as expected
[x] compare last night's perf test send rate to the previous days to ensure system not degraded
[x] Run smoke tests in production once deployed in there. (Steve)

ben851 commented 1 year ago

Ben to release 1.24 in staging today @ 10am and test for downtime during migration

ben851 commented 1 year ago

Ben released 1.24 to staging yesterday, and broke logging
Ben migrated Fluentd to Fluentbit to fix logging, and created a PR for prod release today.
Ben will create a new alert for when logging stops in EKS and Cloudwatch

ben851 commented 1 year ago

I'm having problems with celery scratch impacting celery staging and need to debug. (Second card created)

ben851 commented 1 year ago

Upgraded staging to v1.25. Will aim for production release Thursday am

jimleroyer commented 1 year ago

Waiting on the evicted pods investigation before pursuing with this, as upgrading would destroy/recreate all pods and lose investigation traces.

ben851 commented 11 months ago

Staging at 1.26, prod needs to be upgraded

jimleroyer commented 11 months ago

Potentially upgrading production today or tomorrow.

jimleroyer commented 11 months ago

Blocked until we deploy the DNS propagation.

ben851 commented 11 months ago

Upon further inspection, staging was only at 1.25. Submitted PR for staging and prod to 1.26.

ben851 commented 11 months ago

K8s in staging up to 1.26 smoke test is smoking, logs are logging.

ben851 commented 11 months ago

K8s 1.26 in prod