Proposal: Improved secrets management

[timbod7]

Inspiration:

https://www.fpcomplete.com/blog/announcing-amber-ci-secret-tool/

Implementation:

An assymetric keypair is created in AWS KMS (Key Managment System). The public half of the key is exported to the infrastructure code repo. The private key never leaves (in fact cannot leave) the KMS system.

Secrets are stored in a json file in the infrastructure code repo. The json file is not encrypted, but each individual secret is. An example secrets file:

{
   "encryption_public_key" : "...",
   "secrets": {
       "db_password": "MIB234Sdf@#$5Ssdfsf3234sffSEWR#$TG%^&^&ED#$%#%^DVGXRT%^KLO$%%",
       "auth0_access_key": "...etc...",
   } 
}

A command line tool would be available to developers to add/update secrets in the above file. Note that this tool functions without access to live AWS. With the public key in the file, secrets can be written, though not read. That way developers with access to the git repo can update secrets, revert them, and see when they have changed, but not decrypt and view them.

This file would be mirrored into S3 via the generated terraform.

Server side applications would specify secrets in config via a new ADL types:

union Secret {
   String cleartext;
   String encrypted;
}

In the server config file, with the current approach a secret present in cleartext would appear as:

{
     ...
     "db_password" : {"cleartext",  "xyzzy"},
     ...
}

Going forward, however, the existing camus2 config interpolation would interpolate the encrypted secrets:

{
     ...
     "db_password" : {"encrypted",  "MIB234Sdf@#$5Ssdfsf3234sffSEWR#$TG%^&^&ED#$%#%^DVGXRT%^KLO$%%"},
     ...
}

the server code would be permissioned to access AWS KMS in order to be able to decrypt such secrets as required.

Benefits:

• secrets are managed and versioned with git, with associated audit history
• other than on initial import, devops never see or deal with cleartext secrets
• no error prone manual management of secrets in the AWS secrets manager
• no cleartext secrets present anywhere, except in server memory on use
• encryption private key never leaves AWS KMS
• access to decryption is controlled via AWS IAM permissions and roles.

[barryskal]

Looks great @timbod7 Just one question:

A command line tool would be available to developers to add/update secrets in the above file

Would it be possible to get the tool to also read secrets? I know that we would probably keep copies of some secrets in a project specific bitwarden or similar. However, that will only store 1 copy of the secret. If I check out a really old code version, I may want to see what value was used for the secret.

[paul-thompson-helix]

Would java server code use the AWS SDK to use AWS KMS to decrypt such secrets as required ?

[paul-thompson-helix]

How would DB password work (on the DB side?)