Discovery: Secure key management for Lockbox via KMS

[LindseySaari]

Issue Description

We should look into implementing AWS KMS for secure key management for Lockbox. KMS is a "key management service (KMS) to manage encryption keys and Lockbox / attr_encrypted to do the encryption"

Docs here

---

Tasks

• [x] Research if we should implement kms
• [x] Create tickets for KMS implementation

[LindseySaari]

I'll be gathering my thoughts here and typing up a discovery note. We also have the go ahead to set up auto key rotation with KMS

[LindseySaari]

I came to a roadblock for the Lockbox/KMS feature.

In the instructions here, it says to create a KMS key but it’s not clear if you need to IMPORT a key (possibly the Lockbox.generate_key value) or if you can use one that’s auto generated by AWS.

Unfortunately, imported keys do not support the auto key rotation feature.

I submitted an issue here, but want to wait to hear back

If we do have to do this manually, it wouldn’t be absolutely terrible (and alias values will help), but completely automating this would be awesome. Looking into all the details surrounding this further.

Additionally, see rotating keys manually in the KMS docs

[LindseySaari]

KMS Discovery

How it works

The kms_encrypted gem provides simple secure key management for Lockbox via Amazon KMS. KMS manages the encryption keys while Lockbox takes care of the encryption/decryption.

Setup instructions are easy to follow in the documentation.

AWS KMS

A KMS master key will need to be created in the aws management console and set in our environment. I believe we can set this somewhere in the devops repo. See this section in the docs.

Note: Not sure if we have permissions to create key, so we may need assistance from Ops.

Policies

We will need to create/add IAM policies for encryption/decryption permissions for vets-api server/worker instances.

Logging

It appears that AWS CloudTrail logs all decryption calls. [Alerts](https://github.com/ankane/kms_encrypted#alerting) can be set up for suspecious behavior. This probably needs further discovery.

Testing

the kms_encrypted gem provides functionality for avoiding network calls when testing

Rotation

AWS KMS supports automatic key rotation, but it appears that the automatic rotation happens every 365 days.

"When you enable automatic key rotation for a customer managed key, AWS KMS generates new cryptographic material for the KMS key every year"

Re-encrypting data

However, automatic key rotation has no effect on the data that the KMS key protects. We can possibly set up a rake task that will run after the key rotation to re-encrypt the existing data.

Cost

It appears that each AWS KMS key that you create in KMS costs $1/month. See Service Pricing. We will need one key per environment, plus previous keys may need to hang around for rotation purposes. We will want to set up some automatic deletion process to avoid unncessary costs for old/unused keys. We will additionally have to pay for each encryption/decryption request at $0.03 per 10,000 requests

Usage Costs

Using the calculator, if we have 8 keys, and 8 million requests per month, the existed KMS pricing would be ~$32/month

How often to rotate encryption keys

How often should I rotate my keys? For data-at-rest, key rotations should be done every few months.

Attack Surface

With Lockbox assisting with the encryption of PII, data at rest and in motion will be secured. Data being passed to the database or job queue will be encrypted at the application level before being set in motion.

If the db is compromised, the data viewable by the bad actor will be encrypted and useless to the attacker, but we still need to access the risk if vets-api itself is compromised. If compromised, since vets-api will have encrypt/decrypt permissions in place (for KMS), this will grant the bad actor access to (decrypt) the sensitive data.

Questions: [x] Look into if we can use an AWS generated key [x] If we can, how do we re-encrypt existing data (the current key is in parameter store)

According to ankane's comment here we CAN use AWS generated keys. To rotate existing records, we will just need to define the previous key and remove the notion of a master key