RFE: document building custom Decryptors for external secret management systems

[adamgoossens]

Issue

Currently, PSR uses SOPS to store encrypted values and uses a SOPS decryptor to extract that value on access to the config item.

Many organisations will have third party systems for secret storage (e.g. Vault, Conjur), and these might be the 'official' secret store in those organisations. It would be ideal to be able to use PSR to retrieve and use secrets from these systems, referencing them from config like so (for example): my_vault_secret: VAULT_ENC[/path/to/my/secret]

Possible Solution

This is possible through the ConfigValueDecryptor classes, however it needs to be documented. The documentation would need to cover:

• The base class and required methods
• The flow from encrypted value to decrypted output.
• Use of the obfuscation list to prevent sensitive data from being dumped to stdout/stderr
• How to store and use 'bootstrap secrets' for the Decryptor, e.g. the authentication credentials for a secret store's API.

With documentation in place it becomes easier for additional contributions for other secret store mechanisms.

[itewk]

@adamgoossens i would argue that the entire piont of the DecryptionUtils and ConfigValueDecryptor framework is to support any arbirtrary number of config value decryptors, we simply only have the SOPS implementer right now. So can this RFE be reworked to instead be about having the framework, which we already have, instead be about implementing a specific ConfigValueDecryptor.

Also I think its important to note that SOPS itself can interact with tools like Vault for retrieving the PGP keys, rather then our default approach today of mounting it/them as a secret. If you look under the https://github.com/mozilla/sops#usage you will see instructions for using various cloud providers or Hashicorp Vault as the store for the PGP keys used to decrypt values. We even already have the framework in place in PSR to pass in params to the SOPSConfigValueDecryptor to use something other then file system mounted PGP keys, its just never been tested because didn't have anyone interested in putting in the time to test and write out any necissary instructions. But that is different then using something like Hashicorp Vault to store the values, which would need its own ConfigValueDecryptor.

As far as It does create a need to securely store/reference the credentials used by PSR to access the 3rd party service, however. that is going to be the case no matter the solution we use. The robots will always need some sort of secret to access wherever the rest of the secrets are stored, theres no way around it.

[adamgoossens]

@itewk Perhaps the solution to this is to document how a new Decryptor can be implemented. If we already support it then documenting the process is sufficient for people to build their own implementations. I picked Hashicorp Vault as an example because it was top of mind, but the RFE is about 3rd party secret systems in general. I'll adjust the RFE to make that clear and propose documentation instead.

I agree that SOPS supports 3rd party systems, but I don't see this as a solution - instead of allowing the 3rd party secret system to encrypt and manage my data, I still need to fiddle with PGP keys and encrypt it myself first. That still means that the data is being protected by SOPS, not the 3rd party system, which may raise questions about SOPS encryption/decryption algorithms, FIPS compliance, etc. Better to just let the external system deal with it all and PSR gets the decrypted value.

[itewk]

@adamgoossens sounds good. how about you try to make the HashicorpValueDecryptor and document the process as you go. Unfortunitly we dont have a great place for that sort of doc to live yet. been thinking we need to swap out the psr doc provider to one that allows for arbiratray doc pages to be made rather then the one we use now that just uses the python doc strings. but maybe for now just a docs forlder in the git project with an MD or asciidoc file.