Storing sensitive values in state files

[seanherron]

309 was the first change in Terraform that I could find that moved to store sensitive values in state files, in this case the password value for Amazon RDS. This was a bit of a surprise for me, as previously I've been sharing our state files publicly. I can't do that now, and feel pretty nervous about the idea of storing state files in version control at all (and definitely can't put them on github or anything).

If Terraform is going to store secrets, then some sort of field-level encryption should be built in as well. In the meantime, I'm going to change things around to use https://github.com/AGWA/git-crypt on sensitive files in my repos.

[misham]

What about being able to mark certain fields as secret and then using something similar to the Provisioner Connection to specify the encryption mechanism, including using Vault or AWS KMS or anything else. E.g.

resource "aws_db_instance" "default" {
  name        = "mydb"
  username = "foo"
  password {
    type = "vault"
    method = "transit"
    path = "keys/mydb_pwd"
  }
}

The actual parameters defined in the password block would be specific to the password provider.

During the run, Terraform will use the provided password capability to decrypt and use the value but the state file stores the encrypted value or a reference to it, depending on the password provider.

This way, all of the state does not need to be encrypted and any sensitive data can be secured via whatever process the user likes, similar to how providers work now.

[fernviridian]

Why are my AWS IAM Secret keys being written to my state file? This is ridiculous. I want a flag where I can tell terraform not to write secrets to the state files. We store our credentials in credstash, so there is no need to have them in the terraform.tfstate file. https://www.terraform.io/docs/providers/aws/r/iam_access_key.html#secret

 "resources": {
                "aws_iam_access_key.test": {
                    "type": "aws_iam_access_key",
                    "depends_on": [
                        "aws_iam_user.test"
                    ],
                    "primary": {
                        "id": "<redacted>",
                        "attributes": {
                            "id": "<redacted>",
                            "secret": "<redacted>",
                            "ses_smtp_password": "<redacteD>",
                            "status": "Active",
                            "user": "test"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": ""
                },

[jkodroff]

This was a very nasty surprise for me as the windows_opt_config for the vsphere_virtual_machine resource requires entering fairly high-priv credentials in order to join the machine to the domain.

I understand there may not be an easy solution to this, but at the very least I think there needs to be something in the docs: that this is a known issue and what, if anything, would be a sensible workaround. I think this should be noted both in the the vsphere_virtual_machine page and the Terraform State page.

Just this much could help people from potentially compromising their secrets.

Pardon my ignorance, but would taking a hash of certain sensitive fields fix this issue? Does TF need any more information other than whether the value changed?

Any thoughts on this, @phinze?

[bitglue]

Pardon my ignorance, but would taking a hash of certain sensitive fields fix this issue? Does TF need any more information other than whether the value changed?

What happens if you subsequently add a resource to the configuration which requires the password?

If you hash the password fields, how do you prevent the password from being exposed via other fields in which it has been interpolated?

I think the best solution here is simply to make it clear in the documentation that the state file can be sensitive. That probably means a reference to a section "The state file is sensitive" for every passwordy attribute. If you know its sensitive you can be sure to keep it in a secure location (S3 + KMS?), or encrypt it with any number of excellent file encryption tools. That protects not just your passwords, but a lot of other details about your systems which many kinds of attackers would love to have.

[ProbablyRusty]

In the same way that there is remote state support for Consul, S3, Atlas, etc, I wonder if it might be worth exploring enhancing TF to add Vault as a target for remote state support. Maybe then you could simply supply TF with a Vault token, endpoint, and path to allow for it to read/write its remote state, which would live safely within the confines of Vault.

[ketzacoatl]

This is a fantastic idea that greatly simplifies the annoying and complex issue of how to encrypt/protect secrets... just encrypt the whole thing! I would much rather push my state to vault than to consul, and there's the added benefit of auditing!

[iroller]

There're already some attempts to do that using Vault transit backend. Would definitely love to see Vault provider for terraform variables + state file.

[apparentlymart]

Agreed that putting warnings on the docs for resources that generate secrets is a good interim step for now. We already have some of this in e.g. tls_private_key, but we could do a better job of calling out all the cases that call for state to be protected carefully, and hopefully link to some new central doc somewhere that gives some advice on practical ways to do that.

[chadgrant]

It would really be helpful if you could also add an option to not echo secrets out in plan/apply console output. These get logged in CI systems build logs :(

[zapman449]

Secret values are special. They need to come into Terraform in interesting ways, and they need to be output from terraform (outputs and state files) in interesting ways.

To my mind, password values should always be sha256 or sha512 encoded in the state file. That radically decreases the pain of state file storage, and preserves the principle of least surprise for users.

Regarding outputs for interpolation, secrets have to come into Terraform somehow (before the state file). This means users have two options: 1) mandate that it comes in (tfvars or environment vars, or whatever) when terraform output is called, or 2) force the thing reading from terraform output to read the same input mechanism that terraform uses (shell variables are one avenue for this... tfvars.tf files aren't that hard to parse, etc).

[nickithewatt]

@chadgrant We ran into this issue as well and for the moment we are using terrahelp (a terraform helper) which can mask these values so that they aren't written to log files etc

[Fodoj]

A bit more automated and transparent way to secure statefile is to use something like git-crypt https://github.com/AGWA/git-crypt. You can set per-user permissions for secrets in your repository and encryption and decryption happens automatically when you commit and pull changes. I found it a bit risky to wrap terrahelp with Makefile - people can still accidentally commit secret data to the repo.

[ketzacoatl]

Unless I am misunderstanding something, the git-crypt solution is not really feasible for TF when using remote state (and it doesn't really make sense to use TF in a team without remote state), so securing creds in the state file is still an issue.

[Fodoj]

Remote git repo is a remote state, and git itself is a tool that works really well for teams ;-)

[fernviridian]

I have now revisited this problem for the 4th or 5th time now. This is getting old there is no progress here on an issue from 2014. I can't provision an AWS IAM user without secrets getting leaked into my git repo. Same for RDS and many other services.

I have written a short script to remove secrets from tfstate files that may be useful to others: https://github.com/tgjamin/terraformectomy

[apparentlymart]

@tgjamin it is no longer recommended to put your Terraform state in a git repo. Instead, using Remote State is the suggested approach. This of course won't address the main point of this issue, which is to protect the sensitive values within the state, but it does allow you to use facilities provided by the remote state backend (S3, for example) to control access to the state files and thus limit its exposure.

[fernviridian]

Infrastructure as Code that should not be put in git...I don't like where this is headed.

[johnrengelman]

@tgjamin that's not what @apparentlymart said. He said, it's not recommend to put the state in git. Which is true. The code itself (e.g. the .tf files) can be in git, just the evaluated state/output should be stored in a remote state backend.

[mingfang]

One option is to implement something like Ansible Vault. terraform.tfstate will be encrypted and can be in git. A password is required each time the file is accessed.

[hardboiled]

Is it possible to do something similar to remote state, except for key/values? Basically, I'd like to push a configuration file to an s3 bucket that supports only encrypted files. I don't care about the format as long it supports key/value pairs. Then, in .tf files it might look something like this:

data "remote_config" "credentials" {
  type = "map" /* this could be implicit, meaning flat */
                        /* key/value maps could be the only data-type supported */
  backend = "s3"
  config {
        bucket = "tf-my-bucket"
        key = "credentials.json"
        region = "us-east-2"
    }
}

resource "aws_db_instance" "rds_example" {
   ...
   rds details etc ...
   ...
   username = "#{data.remote_config.credentials["rds_example_username"]}"
   password = "#{data.remote_config.credentials["rds_example_password"]}"
   ...
}

Note as an alternative someone else was loading passwords from a local file. I'd rather it be on s3 though ;).

[Fodoj]

@hardboiled It should be easy to implement with external data source https://www.terraform.io/docs/providers/external/data_source.html It is focused precisely on key-value objects.

[hardboiled]

@Fodoj Actually, while I don't know if this will be my final solution, I instead opted to handcraft a tfstate file and configure it as a remote. Seems to work exactly like people here would want, meaning supporting version control and encryption (since s3 supports those things), with the added bonus of supporting all terraform types, since I'm just manually typing a state file in with outputs :).

state file

{
    "version": 3,
    "terraform_version": "0.8.4",
    "serial": 1,
    "lineage": "",
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "tf-your-s3-bucket",
            "encrypt": "true",
            "key": "credentials/production/terraform.tfstate",
            "region": "us-east-2"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
                "production_rds_main": {
                    "sensitive": false,
                    "type": "map",
                    "value": {
                        "username":"your_org_admin",
                        "password":"your_sensitive_password"
                    }
                }
            },
            "resources": {},
            "depends_on": []
        }
    ]
}

remote configuration in my repo

data "terraform_remote_state" "credentials" {
    backend = "s3"
    config {
        bucket = "tf-your-s3-bucket",
        key = "credentials/production/terraform.tfstate",
        region = "us-east-2"
    }
}

referencing the remote in terraform

resource "aws_db_instance" "production_rds_main" {
   ...
   rds details etc ...
   ...
   username = "${data.terraform_remote_state.credentials.production_rds_main.username}"
   password = "${data.terraform_remote_state.credentials.production_rds_main.password}"
   ...
}

[Fodoj]

That's a really good trick! 👏

[tmichel]

We faced similar issues and ended up using credstash to store secrets. We also wrote a simple terraform provider to integrate with it: https://github.com/sspinc/terraform-provider-credstash

This solves the problem of secrets in variable files but secrets still leak into the state file in plain text.

[hardboiled]

@tmichel Yes, you're right. I checked, and even though I used remote state to pull in the passwords, they were still stored as part of the local state file. However, all my state files, regardless of whether they are my tfstate files or the "config" file I used above as a workaround, are stored on a versioned s3 bucket with encryption. Thus, I don't think this changes the security around my original suggestion.

[coen-hyde]

As @tmichel pointed out since secrets leak into the state file anyway we use self referential outputs via a wrapper script with state stored on s3 with encryption. That is the first time we run some infrastructure with a secret we are prompted for it. Subsequent runs pull the secret from the outputs and set the corresponding env variable with the secret.

[fernviridian]

Update: We have started using git-crypt https://github.com/AGWA/git-crypt for all of our .tfstate and .tfstate.backup files, using our GPG keys to unlock them. This way we can keep all of our state in the git repo we have our other terraform files in, without using remote state storage.

We are also using https://github.com/sspinc/terraform-provider-credstash as we already used credstash for credentials, and this way they are hidden from our terraform files (namely password field of AWS RDS instances)

[odupuy]

We have the original password of RDS in Vault. Some options with S3 are to use acl=private, encrypt=true and possibly your own KMS key but it's not great still. As well, you could use S3 to retrieve the state, then you change the configuration to use the local file system (terraform remote config -backend=local -backend-config="path=.terraform/terraform.state"), run your terraform apply, then obfuscate some values in the state file and manually push the state to S3 in the same location. This being said, it would be nice having a simple solution of some sort.

[dzhus]

How is this better than using git-crypt?

[odupuy]

The local policy is to not have any sensitive values in our Git repositories, even encrypted. We use Vault for this as it has a fine-grained control.

[swoodford]

+1 on this issue, it's a big problem to have IAM secret keys and RDS database passwords in plaintext tfstate files.

[netflash]

@swoodford you can use pgp encryption for IAM keys, https://www.terraform.io/docs/providers/aws/r/iam_access_key.html#pgp_key. But nothing like that for RDS passwords :-(

[mwarkentin]

For RDS passwords, I think you can just set a dummy value in your terraform config and then modify it manually on RDS. Terraform has no way to check that a password hasn't changed, so it seems to work just fine after that without issue.

[swoodford]

thanks @netflash that's great to know it's one option available!

I ended up writing my own script to redact the secret keys and ses smtp password from the state files: https://github.com/swoodford/aws/blob/master/terraform-redact-iam-secrets.sh

It may be helpful to others that want to totally remove them from tf state and git.

[sarneaud]

This won't solve every problem raised in this thread, but it works for a lot of use cases:

I've been experimenting with adding a new no_store key to the lifecycle block, which would be a list of attributes to not store in the state file. It would have to imply ignore_changes. In theory, ignore_changes could be changed to behave this way, too, but that wouldn't give anyone a migration path for dealing with any breakages. ignore_changes could be deprecated.

I'd like some feedback before I put the work into making a production-ready PR, though.

[rayterrill]

Running into this same issue with the windows_opt_config.domain_user_password in the vsphere_virtual_machine resource in the VMware vSphere Provider. I would love some way to say "don't store this to the state file" - a no_store parameter sounds perfect.

[daveadams]

FWIW, this feels like a great opportunity for some Terraform and Vault integration. I'd love to see the ability to specify various encryption providers up front and then within a resource to specify certain fields to encrypt.

provider "transit" {
  alias = "tfstate"
  vault_addr = "https://myvault.example.com"
  key_name = "tfstate"
}

resource "aws_iam_access_key" {
  user = "myuser"
  encrypted_state {
    provider = "transit.tfstate"
    attributes = ["secret"]
  }
}

This would avoid the weird hacky per-resource workaround like the aws_iam_access_key resource has, and would provide a solution for any field in any resource that could be pluggable with any number of encryption providers. KMS, GPG, etc.

[korfuri]

I think the approach of solving this in Terraform core is required, because the sensitive values are not just stored in the state file: they also end up in plan files. This is significant if your workflow involves storing plans, which you're definitely doing if you're using Terraform Enterprise.

With support for sensitive values in TF core, sensitive values should not be stored in the plan file at all. Instead, a step to generate/load the sensitive value should be added to the plan.

[marrik96]

+1 on Terraform core handling secrets better. Not ready for enterprise IMO until then.

[gtmtech]

@apparentlymart Are there any plans to solve this at the moment? Because terraform has no secure data support, other products are being compromised (e.g. the vault provider for hashicorp vault via https://github.com/terraform-providers/terraform-provider-vault/issues/160 ).

[jcmcken]

I'm not sure if I should open a separate issue for this or not, but speaking to the aws_iam_access_key resource specifically, there should no reason to ever store the secret attribute in the state file. The id attribute needs to be stored so that terraform destroy works. But I think if users want to see the secret, they can either use an output or just examine the STDOUT of the terraform apply command. I don't think the resource should store it by default, though

[grimm26]

@jcmcken if you want to see the access key, you need to have that in an output. Outputs are stored in the state file.

[jcmcken]

@grimm26 Yep, I realize that. My point is it's stored in the state file even when you don't specify an output.

[monkey-jeff]

Just another +1 here. I recently opened https://github.com/terraform-providers/terraform-provider-azurerm/pull/2402 to mitigate this issue for service principal passwords in azure as well.

[AndrewFarley]

Guys... 4 years ago this bug was filed, which works out to just about 1500 days. This is like a bad joke, as much as I proudly promote Terraform, mis-steps like this make me really regret it and make it nearly unusable in a strict enterprise/compliance heavy environment.

Please, do something about this as soon as possible. It makes us all look bad. :\

[AndrewFarley]

In a similar note/tone... why does a KMS secret need to store the plaintext AND the ciphertext in the state file? What is the point even of KMS in this scenario? Can't Terraform simply use the ciphertext to look up in the KMS what the value of the plaintext is and use it only in-memory?

Why does it have to be so security insensitive. :(

[daveadams]

I do have to agree that from the company that makes Vault--which has a KMS-like EaaS tool built in that could easily address this problem--this looks embarrassing. The issue of sensitive values in state files is a neverending rough-edge for every team in my company that tries to adopt Terraform into their workflows.

[jmehnle]

I share the frustration about the complete non-addressedness of secrets handling in Terraform, however let me be a bit more productive and suggest another option we have, at least for storing sensitive application/system configuration (such as service credentials) in AWS. At my company we use AWS SSM Parameter Store to store sensitive (and non-sensitive) configuration items, and we encrypt the sensitive ones by hand using aws kms encrypt (or, rather, a nice convenience wrapper) and then add the encrypted value to the definition of an aws_ssm_parameter resource, like so:

resource "aws_ssm_parameter" "neustar_secret" {
  name        = "…/neustar/api_secret"
  type        = "SecureString"
  value       = "AQICAHj...hCoYdxA=="
}

If we ever need to see what the plaintext value was, we use aws kms decrypt.

Here's the AWS KMS string encryption/decryption convenience wrapper.

I know this doesn't work for other cases of sensitive resource attributes in tfstate. Just take the above for what it is.

[pecigonzalo]

That sounds good, but how do you use that somewhere else? Like RDS passwords.

In any case, the secret should be encrypted at rest with most backends. EG: S3 + KMS

So it is secured. Where do you guys see the leak?

On Wed, Dec 5, 2018, 4:09 PM Julian Mehnle <notifications@github.com wrote:

I share the frustration about the complete non-addressedness of secrets handling in Terraform, however let me be a bit more productive and suggest another option we have, at least for storing sensitive application/system configuration (such as service credentials) in AWS. At my company we use AWS SSM Parameter Store to store sensitive (and non-sensitive) configuration items, and we encrypt the sensitive ones by hand using aws kms encrypt (or, rather, a nice convenience wrapper) and then add the encrypted value to the definition of an aws_ssm_parameter resource, like so:

resource "aws_ssm_parameter" "neustar_secret" {

name = "…/neustar/api_secret"

type = "SecureString"

value = "AQICAHj...hCoYdxA=="

}

If we ever need to see what the plaintext value was, we use aws kms decrypt.

Here's the AWS KMS string encryption/decryption convenience wrapper https://gist.github.com/jmehnle/51747d692549b12b7cc245613dc750cf.

I know this doesn't work for other cases of sensitive values in tfstate, such as database passwords, etc. Just take the above for what it is.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hashicorp/terraform/issues/516#issuecomment-444518201, or mute the thread https://github.com/notifications/unsubscribe-auth/AJcviS8rPKxOM_0n1h1zfJwC8rykv18oks5u1-HEgaJpZM4Cz6Ol .

[eedwards-sk]

As I'm currently working on deploying a consul+vault+concourse stack for production use, deployed with terraform, the issue of state file security is highly relevant to me.

Currently, I'm treating the state and plan files as highly sensitive artifacts. Remote state stored in s3 with encryption at rest (AES256), same with plan files. IAM policies can be created to prevent most users from accessing those files.

I started using terraform around v0.6 or so. I've always known the state file was considered sensitive. I didn't have a good solution for it back then.

It's now almost 3 years later, and with all the massive improvements tf has made since, there's still no holistic solve for state files.

While I'm okay with managing it myself, it feels like a break in the chain. Sure I can stand up vault with terraform, but you can get into my vault server just by looking at my state files.