sastels commented 4 months ago

Description

As a Notify team member I need to upgrade the database before it is no longer supported.

WHY are we building?

Support for PostgreSQL 11 will start costing extra $$$ on March 1

Upgrade to PostgreSQL 15

RDS supported without extra cost.

[x] verify / fix the prod tfvars in 1Password (Tuesday)
- [x] rds
- [x] lambda-api
- [x] database-tools
[x] 4 hour load test on staging (Tuesday)
[x] PostgreSQL bug bash (Wednesday)
[x] go through the steps on dev on (Thursday)
[x] reboot RDS on prod (Sat night at 8?)

[x] Step -1: Switch ops genie to Ben and Steve
[x] Step 0: delete the files we use to skip parts of step 1 that are already done: rm /var/tmp/migration_1*
[x] Step 1: remove RDS proxy ./01-PrepareBlueGreen.sh production ~/Notify_env_vars/production.tfvars | tee 2>&1 out_production_1.txt
[x] Step 2: create blue / green ./02-CreateBlueGreenDeployment.sh notification-canada-ca-production-cluster 15.5 default.aurora-postgresql15 | tee 2>&1 out_production_2.txt

[x] run a soak test on prod so we know the length of downtime
[x] Step 3: switchover ./03-SwitchBlueGreen.sh notification-canada-ca-production-cluster | tee 2>&1 out_production_3.txt

[x] Step 4: cleanup blue/green ./04-CleanUpBlueGreenDeployment.sh notification-canada-ca-production-cluster true | tee 2>&1 out_production_4.txt
[x] Step 5: restore RDS proxy ./05-PostStepsBlueGreen.sh production ~/Notify_env_vars/production.tfvars | tee 2>&1 out_production_5.txt
[x] Step 6: terraform PR to set parameter group

jimleroyer commented 4 months ago

Steve created the meeting for the upgrade on next Sunday evening.

sastels commented 4 months ago

instances rebooted Sat 8:21.

9 seconds downtime.
3 api errors, all part of the soak test I was running.
numerous celery errors, enough to trip the critical alarm. Should have switched ops genie, will do during the upgrade Sunday.

ben851 commented 4 months ago

Database upgraded to 15.5 as of Sunday March 11th, 10pm EST