Rotate SECRET_KEY in prod

[sastels]

Description

As a Notify user I need the product to remain secure

WHY are we building?

Want to regularly change our secrets

WHAT are we building?

Rotating SECRET_KEY in prod

VALUE created by our solution

Security!

Steps

https://docs.google.com/document/d/1BkAkz45CVQRGQSNwN018beTebrbTUNpyTpGEaO7Clyc/edit#heading=h.e1u1x5vi3jgm

Step 0: Test the system before doing anything

• [x] POST an email and verify that you receive it and can view it in admin
• [x] verify that the callback was called

Step 1: Rotate SECRET_KEY Assume that the current SECRET_KEY is K1 or K0,K1 Everything has been signed with K1 in the database and in transit K1 is being used for signing K0 (if it’s there) and K1 are used for verifying

• [x] Create a new key K2 using RandomKeygen as discussed above. Because both old containers and new containers will be running during the deployment process we rotate the key in two steps:
• [x] Merge a PR setting the SECRET_KEY to K2,K1
• [x] Wait until the code is fully deployed (wait at least 10 minutes)
• [x] Merge a PR setting the SECRET_KEY to K1,K2

Test:

• [x] POST an email and verify that you receive it, and can view it in admin
• [x] verify that you can view old emails in admin
• [x] verify that the callback was called

Step 2: Resign database fields The database fields for old records have previously been signed with K1. New records are now being signed with K2

• [x] Run the script resign_database.py This should be run on an api pod (so that the appropriate env vars are set and the connection to the database works)

  cd scripts
  python resign_database.py 

  This reports what changes would be made. To actually resign

  python resign_database.py --resign

  This will not resign the notifications. We will let the old notifications move to the history table over the next week.

Test:

• [x] POST an email and verify that you receive it, and can view it in admin
• [x] verify that you can view old emails in admin
• [x] verify that the callback was called

Second rotation:

• [ ] use resign script to verify that nothing is signed with old key
• [ ] Merge PR changing key from K1,K2 to just K2

Acceptance Criteria

• [x] Key rotated
• [x] prod still working
• [ ] original key not in use

QA Steps

• [ ] Tested in a realistic production scenario

[sastels]

PR for K1 -> K2,K1: https://github.com/cds-snc/notification-manifests/pull/1767

[sastels]

Done!

Tested:

• POST to a service with a callback
• smoke tests

[sastels]

Will leave in QA for a day to ensure prod has no issues

[sastels]

looks good. Will wait another week and then rotate a second time. This will remove the old key entirely. Note that waiting is needed to ensure that no notifications are signed with the old key.

[jimleroyer]

We kicked out the old key secret on Thursday in both staging and production environments. We confirmed everything is working as expected. 👍