[Feature] Integration with HashiCorp Vault and its database secret engine

frittentheke commented 4 years ago

Currently the Postgres Operator is able to manage roles it receives from multiple sources (manifest, infrastructure, teams) within the PostgreSQL instance (https://github.com/zalando/postgres-operator/blob/master/docs/user.md#defining-database-roles-in-the-operator).

While this is really quite powerful, we would like to extend further on this and optionally allow to use HashiCorp Vault and its database secrets engine to manage secrets and hand out credentials to individual pods and application instances. Vault adds dynamic credentials and strong audit capabilities. Short lived credentials reduce the risk of secret sprawl and also allow identification of a security breach by having client specific credentials.

At its core this feature would require the operator to register each PostgreSQL instance with Vault and provide superuser credentials in order for Vault to take over the secret managment (https://www.vaultproject.io/api/secret/databases/index.html#configure-connection). To enable Vault to hand out credentials for certains PostgreSQL roles, those might need to be defined as well (https://www.vaultproject.io/api/secret/databases/index.html#create-role), but analogous to infrastructure roles they can be static (https://www.vaultproject.io/docs/secrets/databases/index.html#static-roles). HashiCorp already provides a Golang library for the API and also maintains a Terraform provider (https://github.com/terraform-providers/terraform-provider-vault) using it. Using similar calls towards the Vault API instead of running against the PostgreSQL instances itself would need to be added to the sync loop then (https://github.com/zalando/postgres-operator/blob/master/pkg/cluster/sync.go). Kubernetes Secrets would then not be created anymore as they shall remain inside Vault. Our intention regarding a Vault integration is for it to be an optional and non-breaking extension on the current capabilities by no means a replacement - Not everybody requires/wants to use Vault.

With a few integration points / conditionals in the sync.go loop all other code handling the integration could be held to a different module. A configuration switch to enable this feature and then a Kubernetes Secret holding the settings to configure the Vault provisioning client (to register / delete databases) inside the operator and its behavior would likely be enough and keeps those domains separate while having a joint sync loop to have the PostgreSQL instance as well as the database registered in Vault.

The client side of things, as in requesting a token and then PostgreSQL credentials from Vault is totally out of scope for this feature as this is agnostic from the "source" of PostgreSQL instances anyways. This can be achieved by enabling the application / framework to talk to Vault (i.e. SpringBoot https://spring.io/guides/gs/vault-config/) or to run a sidecar with i.e. ConsulTemplate + pgbouncer keeping all this complexity away from the clients. But this is no job for a PostgreSQL operator, but rather something more in the domain of Vault or the applications themselves.

To cut a long story short, we'd like to discuss if others also see value in integrating with Vault to manage secrets and consequently if you you might accept a PR from our side.

ngerstle-cognite commented 4 years ago

My team is also looking at this- it's not an immediate blocker for using the operator, but we hope that we won't need to spend too much effort trying to work on this going forward. The primary question we've had is how to maintain the operator's access to the PostgreSQL instances, while removing the static password- vault has a decent story on bootstrapping Vault management of access (provide an initial credential then rotate: https://learn.hashicorp.com/vault/secrets-management/db-root-rotation). This can be used for Vault's access, and access managed by Vault, but is largely negated if a static credential for the operator is still present and can't be easily rotated (or perhaps it can, with a vault/consul sidecar?).

frittentheke commented 4 years ago

@ngerstle-cognite thanks for the positive feedback.

And thanks for the hint towards the Operators access to the instances. That piece of code (fetching the superuser credentials from the secret) would need an alternative strategy of fetching that from vault as well. But since we already are a Vault client by that point this is just another invocation of that.

ngerstle-cognite commented 4 years ago

Looking at https://www.vaultproject.io/docs/platform/k8s/injector/index.html, it may not require too much change on the part of the postgres operator, but I'm still thinking about how the bootstrapping process would look..

frittentheke commented 4 years ago

Looking at https://www.vaultproject.io/docs/platform/k8s/injector/index.html, it may not require too much change on the part of the postgres operator, but I'm still thinking about how the bootstrapping process would look..

Yeah on the client side (as I mentioned in my original post) a Vault sidecar or as we already use a consul-template + pg_bouncer service completely handling the Vault part and providing access to the database via localhost:5432 and no authentication to the actual application is a good approach. This is not introducing any changes to the operator though - Apart from keeping it to sprawl static K8S secrets ;-)