Proposal: Storing secret values in a secondary state backend

Current Terraform Version

1.0.0

Background

Secrets management in terraform can be a bit of a hassle. Consider the following resource, an AWS RDS instance with most of its attributes omitted for clarity:

resource "aws_db_instance" "example" {
  name           = "example"
  engine         = "postgres"
  engine_version = "13.1"
  instance_class = "db.m5g.8xlarge"

  password = "???"
}

what's the best way to fill the password field in a way that ensures that the secret stuff remains secret from users? Is there a way for us to have Terraform manage the entire workflow?

Option 1: Static secrets

The easiest solution is to just store the password in plaintext in our terraform repo, and rely on access control on our git repo to keep this secret:

variable "db_password" {
  type    = string
  default = "hunter2"
}

resource "aws_db_instance" "example" {
  # ...
  password = var.db_password
}

This solution is obviously suboptimal - theoretically, anyone with access to the repo our terraform lives in has access to our DB's root password. These two resources (the terraform repo and our DB) don't have the same risk profile, and should have different access control.

Option 2: Passing secrets in at plan-time

Leveling up a bit from the previous solution, we can pass our password in at plan/apply-time:

variable "db_password" {
  type    = string
}

resource "aws_db_instance" "example" {
  # ...
  password = var.db_password
}

and then, when we run terraform, add in the environment variable (or .tfvars.auto file or similar)

$ TF_VAR_db_password=hunter2 terraform plan

Okay, better! The secret isn't stored in our terraform repo, and is harder to leak. This solution still presents a couple of issues though:

The DB password still exists in the terraform state, and can be inadvertently leaked
This workflow means that work has to happen outside of terraform to populate the environment variable that we pass into terraform plan

Option 2b: Managing secrets out-of-band

With this solution, we don't manage the password in terraform at all, instead setting it through some external method.

resource "aws_db_instance" "example" {
  # ...
  password = "dummy value"

  lifecycle {
    ignore_changes = [password]
  }
}

and then, after applying terraform, running something like

$ aws rds update-root-password ....

This has similar benefits and drawbacks to option 2a - we still have to do extra stuff to manage secrets outside of terraform, and the password will still be in state (I think?). Additionally, the terraform configuration for this resource is very unclear - we don't know some other tool manages the password for this RDS instance, just that terraform maybe doesn't manage it.

Use Cases

What I'm really keen for is first-class support in terraform for retrieving secrets from an out-of-band secrets store (Hashicorp Vault, AWS Secrets Manager, AWS SSM Parameter Store, etc), using them in the plan/apply process, and then not permanently storing them in terraform state. These secrets would always be fetched dynamically at plan-time.

These secret values could be considered as a sort of secondary state, but one that terraform never modifies - it only reads, and the "secret state" would be modified out of band, by humans. This shows readers of the terraform config that this secret is explicitly maintained out of band, and where to find it should it need modification.

Proposal

As a super duper draft, I'd like to propose some ergonomics here - this design is mostly to illustrate my preferred workflow. I'm really keen to get feedback here.

(As a note: I'm going to use AWS Secrets Manager as the example source of secrets, as it's what I have the most experience with. Other secrets backends could [and should!] exist. This isn't an endorsement of only using AWS)

New Block Type: `secret_provider`

A secret provider gives terraform the an interface with which to fetch secret information from a given secret backend in the same way as a regular provider provides an interface to work with an infrastructure provider. For example, we might support a secret provider for AWS Secrets Manager

secret_provider "aws_secrets_manager" {
  region  = "ap-southeast-2"
  profile = "mycorp-infrastructure"
  # Support the args that the AWS provider supports
}

New Block Type: `secret`

As a data block is to a provider, a secret is to a secret_provider, and has similar ergonomics. A secret represents one secret value stored in our secret backend, but never gets stored in the state.

Importantly, if a resource references a secret's value, instead of having the secret's value stored in the terraform state, we'll store a reference to the secret. The secret's actual value will only get pulled in during the apply cycle.

secret "aws_secrets_manager_secret" "db_password" {
  name = "/db/passwords/example"
  # And any other args supported by secret manager
}

resource "aws_db_instance" "example" {
  # ...
  password = secret.aws_secrets_manager_secret.db_password.value
}

I'm intentionally not proposing a resource equivalent for a secret_provider - I think like with backend configuration, Terraform will have to just assume that secrets have been created out of banc, and complain if they're not present.

References

https://github.com/hashicorp/terraform/issues/516

hashicorp / terraform