Secrets management

[krogebry]

Overview

In order to properly secure secrets data we want to start using Vault.

There are essentially two ways to map secret data into a container using Vault:

• Using ESO: https://external-secrets.io/latest/
• Using Vault annotation: https://developer.hashicorp.com/vault/docs/platform/k8s/injector/annotations

The vault annotations is a fine way to go, but there's a draw back that most people don't realize that you only really get when you're doing this in production. Occasionally, the vault system will fail, and when that happens, none of your pods will come up because they're dependand on the vault service for key data.

On the plus side, this ensure that the only place your secret data is stored is in Vault. That's great and all, until your cluster shits the bed and you can't get anything to work.

This is where ESO comes in. The External Secrets Operator allows you to map things in a certain way where you end up with a k8s secret with your secret data. So, on one hand that does break the model where you have data in vault ( or AWS::SM, or whatever ), and in a k8s secret in the namespace. But it's a secret, and therefor it's protected more than just a normal object in k8s.

This also means two things:

• Containers continue to work if Vault ( or whatever ) goes down, because the k8s secrets are objects in the namespace.
• ESO checks for updates every hour, so if something does change ( key rotation, for example ), the new vaules can be written into the env automatically.

Even if we're using a local vault service, I'd still prefer to do the "long way" with the ESO as to ensure that we have a functional environment. I would also suggest that we use the annotations way as well just get practice with how that works.

DoD

• Deploy vault to the cluster using terraform
• Ensure everyone can see the vault ( hopefully we're able to use the TS operator to create a vault node on the network which points to an Ingress for the service )
• We might have to give everyone admin credentials for the vault service, but that should be fine, we can use lastpass or whatever for that
• Begin teaching services how to interact with Vault so that we can map values into containers
• Deploy ESO
• Configure ExternalSecrets to map to vault values for services

[mistyayn]

@krogebry am I on the right track with these for getting vault to run in the cluster?

https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-raft-deployment-guide

https://developer.hashicorp.com/vault/tutorials/operations/codify-mgmt-vault-terraform

[krogebry]

Yeah, more or less.