Rotate database master password in production

[sastels]

Description

As a Notify user, I need secrets to be secure and occasionally changed

WHY are we building?

Want to rotate the database master password

WHAT are we building?

Moving the app to a new user and rotating the master password

VALUE created by our solution

password has been changed

Tasks

• [x] Create a user app_db_user (can use the scripts in the attic to create the user).

• [x] Add the user to the RDS Proxy

• [x] Verify on the jump box that the new user can connect to the database

• [x] Change values in manifest repo to be generated from app_db_user:

  • POSTGRES_SQL
  • SQLALCHEMY_DATABASE_READER_URI

• [x] Verify that the app and Hasura still work (also: smoke test api k8s)

• [x] Change secrets in terraform to be generated from app_db_user:

  • PRODUCTION_SQLALCHEMY_DATABASE_READER_URI
  • PRODUCTION_SQLALCHEMY_DATABASE_URI

• [x] Run the merge to production script

• [x] Redeploy the blazer service (so the new variable gets picked up)

  • Go to the blazer service / update
  • Select “Force new deployment” and click “Update”
  • Go to deployments tab and wait

• [x] Verify that blazer and api lambda still work

• [x] Add the new terraform variables in LastPass

• [x] Change the master password!

  • change the terraform secret PRODUCTION_RDS_CLUSTER_PASSWORD

• [x] Run the merge to production script

• [x] Verify that app / Blazer / Hasura still work

• [x] Add the new value in LastPass

Acceptance Criteria

• [x] app moved to another user
• [x] master password rotated in production
• [x] new db password and terraform variables are in LastPass

QA Steps

• [x] verify app still passes smoke tests
• [x] verify that admin still works
• [x] verify Hasura still works
• [x] verify Blazer still works