KMS Key Rotation - 1. Research

[holdenhinkle]

ticket details to come...

Issue Description

What details are necessary for understanding the specific work or request tracked by this issue?

---

Tasks

• [ ] What work is necessary for this story to be completed?

Acceptance Criteria

• [ ] What will be created or happen as a result of this story?

  How to configure this issue

• [ ] Attached to a Milestone (when will this be completed?)
• [ ] Attached to an Epic (what body of work is this a part of?)
• [ ] Labeled with Team (product support, analytics-insights, operations, service-design, Console-Services, tools-fe)
• [ ] Labeled with Practice Area (backend, frontend, devops, design, research, product, ia, qa, analytics, contact center, research, accessibility, content)
• [ ] Labeled with Type (bug, request, discovery, documentation, etc.)

[holdenhinkle]

Google doc - https://docs.google.com/document/d/1kJBpyB7IdOxV-tgs2VaqbtHf1nGV96v0OiaNWh4Y02w/edit#

[holdenhinkle]

Main issues:

• What if a job fails? Do we have to start over or is there a way to pick back up where we left off?
• Can we automatically divvy up the table records amongst n number of GitHub Actions jobs/Sidekiq workers?

Issue #1: What if a job fails?

• Add the following field to the tables that need to be encrypted:
• encryption_updated_at, datetime
• when updating the “encryption_updated_at” field sure to do a save(touch: false) to avoid touching the timestamps and causing unwanted behavior
• When updating the table
  • Select records where encryption_updated_at > 11 months from now or NULL
• Update encryption_updated_at as records are re-encrypted

Issue #2: Can we automatically divvy up tables/records amongst n number of Sidekiq workers?

• Given a list of tables, select n number of records, iterating through the tables until n records have been selected or until there are no more records to be selected (records that need to be updated)
• Divide the records amongst n number of workers
• Each worker processes the given records
• Repeat until all records have been updated

---

Lindsey and I met yesterday (11/7/2022) and discussed the ideas outlined in the above linked document.

A couple of main points came out of it:

• Because we’re not able to access the vets-api db when vets-api is checked out and is running in GitHub Actions, we can’t use GitHub Actions. Sidekiq, however, does have access to the DB in each environment. We’ll use Sidekiq.
• Also, instead of running this job on a schedule using CRON, it seems to make more sense to run this as a rake task in each environment, manually. That way, we can first run it in dev, and then staging, to make sure it’s working correctly, before running it in prod. Most importantly, it will allow us to monitor it while it’s running (there’s a chance the job won’t be monitored if it’s kicked off automatically).
• Lindsey suggested that I look into Sidekiq Batches
• Include a way to cancel the job

---

Links

Sidekiq Batches - https://github.com/mperham/sidekiq/wiki/Batches#overview

---

Pseudocode

Class Batcher

initialize

Create new sidekiq batch instance

• set batch description
• set batch callback

batch_records

• set var records to empty array
• call #get_model_names
• iterate over model_names
• for each model name,
  • query the table for records where
  • encryption_updated_at is null or is older than 11 months
  • limit n numbner of records (perhaps start with 1000)
  • add selected records to the records array
  • if records array is less than 1000
  • call next to query the next model's table
  • set the limit to 1000 - records.length
  • continue this process until 1000 records are select or until you've iterated through all models
• iterate through all records, sending each record to KMSKeyRotator.new.perform_async(record)
• if there are no more records, exit

private

on_complete

this method is a callback that's called when the current batch is complete we can log the results of the previous batch this method also acts as a loop to keep the job running by creating another instance of Batcher => call Batcher.new

get_model_names

returns an array of model names that we want to rotate the keys for

~ - - -

Class KMSKeyRotator

Includes Sidekiq::Job

perform(record)

• re-encrypt record and update encryption_updated_at

[holdenhinkle]

Research and Solutioning is kinda blurred.

I'm planning on opening the Implementation ticket tomorrow and coding this up.

[holdenhinkle]

I'll also write a rake task that will kick this off.