Storing sensitive values in state files

[seanherron]

309 was the first change in Terraform that I could find that moved to store sensitive values in state files, in this case the password value for Amazon RDS. This was a bit of a surprise for me, as previously I've been sharing our state files publicly. I can't do that now, and feel pretty nervous about the idea of storing state files in version control at all (and definitely can't put them on github or anything).

If Terraform is going to store secrets, then some sort of field-level encryption should be built in as well. In the meantime, I'm going to change things around to use https://github.com/AGWA/git-crypt on sensitive files in my repos.

[dentarg]

I think the announcement from yesterday is somewhat relevant to this issue: Introducing Terraform Cloud Remote State Management

[aitorpazos]

Would it make sense to provide predefined infrastructure initialisation templates as part of terraform init command? Sounds like depending on the environment you are in (hobbyist, AWS-only user, Terraform enterprise user, etc.) different users would be happy with different approaches. For example, I set up a KMS + S3 + DynamoDB template that allows to restrict the access to those secrets only to the terraform runner roles. I guess we are all re-inventing the wheel in the AWS front. Feels like this infrastructure provisioning with best practices (setting up storage encryption keys, restricting storage access, creating storage resources, etc.) built-in would be a nice to have and not hard to implement for those backends based on cloud providers.

[ringods]

This is a module used for a lot of customers of Skyscrapers: https://github.com/skyscrapers/terraform-state

[daveadams]

Just to clarify my own use case here: We use S3 with encryption-at-rest and access rules to safely partition who can access state files that might have sensitive values in them. But I'd really like to be able to share all our state across our engineering organization, but without having to share all secrets potentially embedded in those state files. In any particular tfstate file, 99% of the info is fine to share across the org, but I don't want the root RDS password floating around. Or an IAM secret key. The IAM access key provider built its own custom encryption for that single field, but this seems like something Terraform should provide at the top level. I should be able to say on a resource "encrypt these fields with this encryption provider", and I should be able to point to specific KMS keys or Vault transit keys, or PGP, or whatever, just for the fields I need to protect, not for the entire state.

[eedwards-sk]

@daveadams the challenge with that approach is that secret values get passed all around a terraform configuration, often in ways that leak them to the output.

e.g. if your secret keys need to go into a templated user data file, you'll probably end up with the raw secret in the state file as a part of the user data template, even if the source variable itself is encrypted

[ketzacoatl]

@eedwards-sk that's true, but we should not delay or contort the simple case for practices which we know we should be improving upon and doing differently.

EDIT, to share that terse message more clearly: We ought to use a tool such as vault or credstash for what those use cases you describe, but we still need Terraform to consider secret/sensitive values as first class-citizens, eg when we use the IAM resources to create and manage some key which we hand off to an automated agent or vault or which we would want to share with an operator. Terraform would need to support encrypting and decrypting values with our GPG keys, or keybase, etc, and would need to support passing around and handling these encrypted secrets internally (between modules and graphing/planning/etc). Encrypted values could also be written to outputs and made easily accessible to operators via keybase/GPG integration.

[ketzacoatl]

I think the announcement from yesterday is somewhat relevant to this issue: Introducing Terraform Cloud Remote State Management

It's relevant, I'd agree. Though I think it's worth pointing out that, without support for storing secrets internally, pushing your TF statefile to hashicorp's cloud is pushing those secrets up there too, unencrypted. Obviously, Hashicorp would use encryption for the statefiles, and would be sensible about key management, but it's easier to trust our own S3 buckets until Terraform supports secrets as encrypted values.

[daveadams]

Right, I realize there are still risks to embedded secrets... but right now there are zero tools to manage that risk. This seems like a perfect opportunity for integration with Vault, as well. I get that it might be low on their priority list compared to other things, but I would like to hear that they had some plan for addressing this risk.

Everything I've heard from Hashicorp has been along the lines of "oh, just encrypt the whole thing", but now that they are pushing their cloud for storing state, the risk of unencrypted inline secrets is suddenly a lot bigger. If I could inline-encrypt sensitive values before they were stored in Hashicorp's cloud, I might consider using it. But without this capability, it's a no-go.

[pauldraper]

@daveadams is absolutely correct. "oh, just encrypt the whole thing" is something can be trivially done already and is not a solution.

Currently, any operation with terraform state (i.e. everything with terraform) require super duper secret access. There's no way to broaden the usage of Terraform, without broadening the access to secrets.

A fundamental principal of security-minded systems is separation secret and non-secret information. If everything is non-secret, it is insecure. If everything is secret ("just encrypt the whole thing"), then it is often impossible to use.

Ideally, terraform would be useable without always having access to secrets. Three possibilities:

1. Don't store or diff on secret information. (Simply making ignore_changes not appear in the state could solve this. #20349)
2. Store secrets in a separate provider, and optionally merge them back into the state.
3. Store secrets encrypted in the state file, and optionally decrypt them with the provided key.

[robzr]

Seems like a really easy way to fix 90% of the problem for people using Terraform in CI/CD flows and have no way of preventing sensitive input variables from making it to log output would be to just mask or hash it in the output. If an input variable is marked "sensitive" then filter the attributes in the log output for the value and mask them.

And why not just hash sensitive values in the state. If the state just exists to compare and find differences, why would it need the actual values.

[acdha]

Having just had to jump through some extra hoops to be able to use Terraform on a project, it seems like a large fraction of my problem would have been avoidable with two features:

1. Something equivalent to the CloudFormation resolve feature where the value stored in the state would be a reference (e.g. an AWS Secret's ARN) but the actual API call would use the lookup result.

2. A per-project opt-in feature which could be used to trigger errors any time a secret is used in a way which would leak it into the state so the various providers could flag sensitive data and resource attributes and organizations which require modern security levels could be confident that a simple mistake is no longer possible.

[klebediev]

hundreds if not thousands of companies are affected by this security issue. @apparentlymart since the issue in open for more than 5 years may I ask about: 1) whether it's going to be fixed and if the answer is "yes": 2) how it will be fixed 3) when it's planned to be fixed

[pauldraper]

Something equivalent to the CloudFormation resolve feature

For anyone using AWS, one solution is to create a CloudFormation stack for any resources requiring secret paramters.

You then store the secret in SSM/Secrets Manager and have the CloudFormation stack resolve that value.

Pros:

• Secrets never appear in your Terraform state files.
• Changes happen when secrets change.

Cons:

• It's an extra layer of indirection.
• It only for AWS resources.

[pecigonzalo]

Isnt that the same tho? SSM is encrypted with KMS, as should be your S3 bucket.

[bgshacklett]

Access to secrets can be limited significantly more than access to a terraform state file in S3. Following the above example, it's possible to grant access to a given secret to an execution role which will be used by CloudFormation, while the user initiating the action would not have access to that same secret.

[pecigonzalo]

Similarly you can grant access to the system running terraform but not to users. If you are running terraform to prod manually, I would be more concerned with that.

[acdha]

In addition to what @bgshacklett noted, SSM/Secrets are also independently logged: if you use a state file in S3, every operation has to be logged as potential access to secrets (e.g. terraform plan has to be restricted, too) whereas SSM/Secrets would only have retrievals logged by the services which actually use them.

[pecigonzalo]

@acdha That is true, dont get me wrong, I think it would be great to have a feature like this, but I dont think its the "big problem" some comments make it seems. Its more of a quality of life, not a big security risk.

[bgshacklett]

@pecigonzalo There are still a lot of companies out there that are deploying to prod by manually copying files over FTP. The steps towards best practices are often incremental and having the flexibility to control secrets separately from the rest of the TF state could be very beneficial for those who aren't capable of making the leap to a full integrated deployment pipeline.

Also, I would consider quality of life changes quite important when it comes to security. They can often mean the difference between something being implemented properly or not due to the compressed timelines and shifting priorities that so many in this industry work under.

[pecigonzalo]

There are still a lot of companies out there that are deploying to prod by manually copying files over FTP

But do you think for those companies their biggest concern should be the encrypted S3 files holding their state? If you are injecting the secrets manually at runtime, because you are applying manually, then you already have access to the secrets.

Dont forget that CloudFormation, which was mentioned before, does this by retrieving the secrets at runtime, as Terraform Cloud would do and then problem solved. The reason CF can do this is because they run the apply and etc remotely, not locally. You can achieve a similar workflow by putting the secrets in SSM, not having access to the S3 bucket (similarly how you dont have access to the storage of CF state) and running in a remote place (Terraform Cloud, CI, whatever). For remote operations check: https://www.terraform.io/docs/backends/types/remote.html

Again, I agree, I think this feature will be a great enhancement, but I believe we are coming at it from the wrong angle. What is being asked here is for local runs to not be able to access those secrets, which is more than complicated given you want your local machine to evaluate and compare state to be able to apply a plan, but to do so without access to secrets, which are part of what it might need to change or evaluate.

[acdha]

But do you think for those companies their biggest concern should be the encrypted S3 files holding their state? If you are injecting the secrets manually at runtime, because you are applying manually, then you already have access to the secrets.

Think about the time scale and who has access: under the S3 method, anyone who ever gets access to that bucket or any system where Terraform has been used has a copy of all of the credentials. Someone on your ops team has a backup compromised or installs the wrong NPM module and something they touched 3 months ago has been leaked.

In contrast, if you were running Terraform locally and it retrieved secrets on demand without persisting them in the state file, the secrets are only at risk if you're compromised when it runs. This also means that things like CI processes can do things like plan without ever having access to those secrets. This really just goes back to the basic security principal that each copy entails risk: since all of my secrets are stored in an encrypted store which can be queried at runtime, saving a static copy somewhere else in a separate system doubles the number of things which have to be secured.

[pecigonzalo]

Think about the time scale and who has access: under the S3 method, anyone who ever gets access to that bucket or any system where Terraform has been used has a copy of all of the credentials.

Terraform does not keep a copy of the state locally when using remote state as far as I recall . Again you are comparing what CF does by running remotely, to what you do running locally. Just use Terraform Cloud and you can compare apples to apples. If users are keeping a local copy, they might as well be storing secrets locally on a .envrc file or something... Or even storing AWS creds directly in ~/.aws/credentials not using aws-vault or any for of aws assume-role.

In contrast, if you were running Terraform locally and it retrieved secrets on demand without persisting them in the state file, the secrets are only at risk if you're compromised when it runs

If a user can retrieve secrets at runtime for apply, the he already has access to the secrets. You should be using AWS assume roles, or aws-vault to only show AWS creds at runtime. This is just beyond Terraform, this is a general AWS good practice.

This also means that things like CI processes can do things like plan without ever having access to those secrets.

How? How can you plan without knowing if the secret changed?

saving a static copy somewhere else in a separate system doubles the number of things which have to be secured.

I agree with this part, but again, is not the massive issue some users here are making it seem.

EDIT:

Im certain that in many cases, the password that ends in the state file could be either avoided by just making the resource point to the SSM or passing strings encrypted with SSM that the resources can decrypt at runtime. Check the discussion here: https://github.com/hashicorp/terraform/pull/874#issuecomment-373365963

If you read the comments in this issue, you will find https://github.com/hashicorp/terraform/issues/516#issuecomment-320825592 and https://github.com/hashicorp/terraform/issues/516#issuecomment-379078312 the only ideas to fix this.

[daveadams]

One use case that I think is being overlooked here is state sharing by different teams. For example, we have an S3 bucket in which we store state for thousands of TF projects maintained by dozens of teams. Some of those projects fetch the state of other projects from that same bucket, and use outputs of one project as input values.

But, we don't want secrets stored in all the projects to be readable by every teams. We might want them to be able to look up an RDS DB's DNS name, but not its root password, for example. And we don't want to have to worry about if state gets copied locally for troubleshooting or for any number of other reasons.

This is not achievable by an all-or-nothing encryption scheme where access to anything is access to everything. Remote execution doesn't solve this problem. We would still want a barrier between different projects and teams. Executing an apply on project 1 that needs some tidbit of output from project 2 shouldn't imply that project 1 can read all of project 2's secrets, whether the apply is being run by a human or by a CI system.

[pecigonzalo]

But, we don't want secrets stored in all the projects to be readable by every teams

You can achieve this by storing the output somewhere else, like SSM even instead of querying the state. This is a known pattern used in many projects. Check https://github.com/cloudposse for example.

Remote execution doesn't solve this problem.

Why? How is this different to how it works on CF?

[daveadams]

So @pecigonzalo, yes, if we added the extra complexity of a three new layers of complexity around using Terraform (a second external storage system for "safe" shared state, plus the wrappers to implement it, plus an external system for running Terraform), we could achieve state safety. It's true that you can solve pretty much any software problem with wrappers and indirection.

But this ticket is not asking for ways to work around Terraform's limitations. We are hoping to get a feature built into Terraform that will avoid the need for all that extra complexity, and make secure state easier to achieve in a lot of scenarios that are different than yours.

[pecigonzalo]

@daveadams Using the state across teams as an interface is IMO a miss-use which can even bite you in the future as now your terraform state is locked as the interface between 2 teams. This is like 2 separate services talking to the same database. The state is primarily the state of terraform and not an access controller between teams, if you want to achieve proper access control, you most likely are going to need a tool that is intended to do that.

Terraform must store state about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.

• https://www.terraform.io/docs/state/index.html

In any case, you have not answer how this is not solved with remote state and using remote execution, given the individual users no longer would have access to state files.

[pecigonzalo]

I was just thinking, maybe a better feature for this given the mentioned use cases would be to optionally separate the output to its own files/storages.

This I recon would be much simpler to implement and would mitigate a lot of the concerns.

[pecigonzalo]

Without a native provider you could do something like (pseudocode):

Write:

locals {
  sharedoutput = {
    address = "asda"
    someother = "thisthat"
  }
}

resource "aws_s3_bucket_object" "examplebucket_object" {
  key                    = "someobject"
  bucket                 = "somebucket"
  content                 = "${jsonencode(local.sharedoutput)}"
  server_side_encryption = "AES256"
}

Read:

data "aws_s3_bucket_object" "bucket_ouput" {
  bucket = "somebucket"
  key    = "someobject"
}

locals {
  bucketinput = "${jsondecode(data.aws_s3_bucket_object.bucket_ouput.body)}"
}

Which should work similarly to retrieving state, but allows you to customize what to put there or how to share it. Its somewhat similar to my SSM suggestion, maybe a bit simpler given its a single file.

If this is provided as a native resource to get/set then it should be quite straight forward. Altho, this is quite simple to do to be honest.

[daveadams]

For remote execution, if you can refer to other remote state in your project, you can (intentionally or accidentally) leak secrets from it.

[daveadams]

I'm honestly confused why you are pushing back so hard on this @pecigonzalo. We all understand there are workarounds. We are all using workarounds already, which is why this feature is being requested.

[pecigonzalo]

Because I believe a lot of the urgency pushed here is unfounded. People compare this to CF when on CF they run remotely, without complains apparently, but here they want to run locally and not have access to the secrets.

Many of the use-cases, like the one about output sharing you described, are not really a big problem or not a problem of terraform state management and do not belong on this issue or are not a problem at all, as explained and shown in my example.

To add, how is my split S3 output example a lot more complex than just sharing state? Its 3 lines of code and fairly clear and decoupled from terraform itself. IMO sharing state files is a lot more complex and a bad practice if applied anywhere, because you cannot mutate the interface or change buckets, or config without affecting downstream users. While in my example you can easily mutate and evolve the interface. Its as I said before, as a bad practice as the example of 2 separate services accessing the same database.

[pecigonzalo]

Just another clarification, I am in favor of the feature, not because it prevents user access when running locally or because we need to share outputs, but to avoid secrets being copied outside of their sources when used in combination with lets say, Vault or other similar tools. Which is a much smaller problem than mixing all this things together.

[acdha]

@pecigonzalo I think it would be helpful if you were more careful to respect that not everyone works the same way on the same projects with the same security footing as you, and that's perfectly valid. The thread is full of valid use-cases and given that this is a security consideration, I would especially suggest considering how the default behaviour being less secure could shape an organization's view of Terraform as they're just getting started, especially given that this does not happen to anyone using CDK which is secure by default:

CDK RDS:

By default, the master password will be generated and stored in AWS Secrets Manager.

My suggestion back in October to have Terraform use Secret versions for the state record and only retrieve the value right before performing the actual API call to create/update would not be a substantial change, would allow lower-privilege plans, would work both locally and remote with any backend, and would allow the documentation for popular services like RDS to have a clear, simple “Always use {Secrets,SSM} to store the password” message and a runtime warning/error if you did anything else. Given how many times I've seen this crop up on different projects, that seems like a meaningful real-world security win even if it doesn't solve everything.

[pecigonzalo]

@acdha I do respect that is the case, that is why Im trying to point those users in what I believe is right direction. I do not agree the thread is full of valid use-cases, many of them are indeed trying to do something that they should be doing some other way. This is not a minor change, its quite complex and because of that we should limit the scope and be clear on what we need and want to archive from this.

You suggestion in October is a good approach, but the main reason that works is because as mentioned multiple times, CF runs remotely, we dont actually know if somewhere in a private state CF stores a sha or something to be able to tell if the secret changed. While it I agree it would be good to have a simpler way to reference this resources, you can do this today, by using SSM data resources, and not applying locally.

[acdha]

You suggestion in October is a good approach, but the main reason that works is because as mentioned multiple times, CF runs remotely, we dont actually know if somewhere in a private state CF stores a sha or something to be able to tell if the secret changed.

Why can't Terraform store the version number of the SSM Parameter or Secret and trigger an update any time the specified / queried (in the AWSLATEST case) changes? Both services use immutable versions so it's guaranteed to change any time the value changes.

[pecigonzalo]

Yeah I already said I think its a good approach, but you need something that translates to other providers as well and it also needs the user to query the parameter on plan, so you still need access to the secret.

[jakubigla]

Why can't we have something similar to lifecycle -> ignore_changes like ignore_state_file_values?

If the user specifies a key, the value won't be stored and terraform plan will not be checking this key for changes. Just like lifecycle -> ignore_changes. This could be done on a provider level too.

[bgshacklett]

What is being asked here is for local runs to not be able to access those secrets, which is more than complicated given you want your local machine to evaluate and compare state to be able to apply a plan, but to do so without access to secrets, which are part of what it might need to change or evaluate.

Logically speaking (tech debt and refactoring aside), I see two ways around this:

1. Remove the evaluation of secrets from Terraform's scope; just pass them through. Update the secret every time unless the user configures the lifecycle rules to ignore changes on that value. In the long term, perhaps a method for hinting to Terraform that the value should be updated could be passed in from the outside.
2. Ensure that secrets are hashed prior to any persistence or outputs other than resource configuration. If it's truly necessary for Terraform to know whether a value has changed, it's just as simple to compare two hashes as it is to compare the value of two secrets.

Either of these would need to be opt-in, obviously, but that's a fairly standard pattern now, and Terraform already has a reasonably complex type system which should be able to handle additional metadata. Providers would require updates, but this might be doable in a largely backwards compatible manner, resulting in those updates being limited to cases where a resource's value might be a secret.

On another note, the question of local or remote runs continues to be brought up. Aside from auditability, I don't see how it matters from a security perspective. @acdha made a very good point, previously:

Think about the time scale and who has access: under the S3 method, anyone who ever gets access to that bucket or any system where Terraform has been used has a copy of all of the credentials.

This is absolutely accurate. The state file doesn't need to be stored as a copy locally, because the system has full access to the bucket. Even if we assume that Jenkins, CodeBuild, Azure Automation, ad nauseum, is running the builds and is running in an environment with no direct access via SSH or other methods, it's still quite easy for anyone who has capabilities to run Terraform jobs to pull down a copy of that state file from S3. The whole point of Jenkins, or any of these CI tools is to run commands. Actually, let me rephrase that: "the main purpose of a system like Jenkins is to execute arbitrary code on behalf of the end-user!"

Running in an automated system does not make you more secure. At best, it makes things repeatable (the original goal of CI/CD) and more visible, so that exploits are, hopefully, seen and tracked. At worst, it gives you a false sense of security leading to unintended data exfiltration.

[zerkms]

Ensure that secrets are hashed prior to any persistence or outputs other than resource configuration. If it's truly necessary for Terraform to know whether a value has changed, it's just as simple to compare two hashes as it is to compare the value of two secrets.

@bgshacklett not all values can be obtained on the second run. Eg: aws iam access key secret.

So it won't work.

[daveadams]

It's worth pointing out that if the infrastructure is in place in the Terraform code to provide a means of hashing potentially secret values, then you'd have all the pieces you'd need to make per-field encryption via Transit or PGP or KMS or whatever. Encrypting and decrypting values in any of those systems is very easy to do. The hard part here is that Terraform would need to provide a common mechanism for doing so, rather than the very rare one-off provider resource that has some resource-specific methods, eg aws_iam_access_key.

[pecigonzalo]

Why can't we have something similar to lifecycle -> ignore_changes like ignore_state_file_values?

I would be good but I dont think its that simple, because if I remember correctly, ignore_changes still stores the value as it required to resolve values for other dependent resources, it will just stop applying and storing subsequent changes. Check https://github.com/hashicorp/terraform/pull/15797 as well for more info on this topic.

Remove the evaluation of secrets from Terraform's scope; just pass them through. Update the secret every time unless the user configures the lifecycle rules to ignore changes on that value. In the long term, perhaps a method for hinting to Terraform that the value should be updated could be passed in from the outside.

Wouldnt that mean you need access to the secret every time anyway?

Ensure that secrets are hashed prior to any persistence or outputs other than resource configuration. If it's truly necessary for Terraform to know whether a value has changed, it's just as simple to compare two hashes as it is to compare the value of two secrets.

• How does this work if I add a new resource afterwards that uses that secret? Terraform needs to know how to "unhash" it so it can send it to the resource.
• To apply/plan you needs to retrieve the secrets on the runner, so if its local, you need access to the secret anyway.

On another note, the question of local or remote runs continues to be brought up. Aside from auditability, I don't see how it matters from a security perspective.

It matters because the users no longer have direct access to the secrets, and it keeps being brought up because people keep comparing local Terraform runs to remote CloudFormation runs, which is not comparable.

Think about the time scale and who has access: under the S3 method, anyone who ever gets access to that bucket or any system where Terraform has been used has a copy of all of the credentials.

This is absolutely accurate.

I do no think it is, systems that ran Terraform do not store a local copy of your state when using remote state. It just keeps this for example.

{
    "version": 3,
    "serial": 1,
    "lineage": "UUID",
    "backend": {
        "type": "s3",
        "config": {
            "bucket": "BUCKET",
            "key": "KEY",
            "region": "REGION"
        },
        "hash": HASH
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

The "modules", etc, section remains empty.

You can also limit access fairly simply, by using different KMS keys or buckets, who access which states like you would do for secrets. The only problem here would be if you are using remote state to populate input to other terraform things. Which I think would be solved by my previous example (which could indeed be made simpler with a native resource to bundle that logic across providers)

Running in an automated system does not make you more secure. At best, it makes things repeatable (the original goal of CI/CD) and more visible, so that exploits are, hopefully, seen and tracked. At worst, it gives you a false sense of security leading to unintended data exfiltration.

I disagree, it depends quite a lot on the implementation, its not about an automation system or not, it is about restricting the systems that have access to the secrets and how they can access those secrets, otherwise your blank statement would apply to all your systems that have access to secure content, like your systems using said secrets.

If your main concern are internal intentional hacking, then IMO allowing people to run arbitrary code on your CI/CD or whatever system with escalated privileges is your main problem in any case, not Terraform in particular. The automation system already has access to create and modify resources (in order to run terraform apply) as well as passwords, etc. so it has access to retrieve those values or change them to allow access, or even destroy them in many cases.

Many Automation systems already have protections for this kind of situations, like preventing certain commands, or only allowing to pass in parameters and using the pre-defined run configuration only. Just in case, this is the same case with CloudFormation, anyone that can push cloudformation could modify it to store the secret somewhere or print it out and then retrieve it.

It's worth pointing out that if the infrastructure is in place in the Terraform code to provide a means of hashing potentially secret values, then you'd have all the pieces you'd need to make per-field encryption via Transit or PGP or KMS or whatever.

But what do you gain here? If you still can decrypt the secrets, then is no difference.

---

The best solution for most resources is IMO outside of terraform and is to have the actual resources be able to retrieve from pointers to secrets (like ECS does with SSM/SecretManager or RDS with SecretManager, Kubernetes with many providers, etc) this ensures no tools has access except the secret storage and resource.

There are other critical cases like aws_iam_access_key or tls_private_key which are a bit more complicated, as it has to store or present the output somewhere. aws_iam_access_key has pgp_key for encryption of the output, but its still stored in the state.

I think it would be good to list the cases that are a "priority" or a problem as not all resources or sensitive content is equal, so we can go over them and evaluate possible fixes. For example:

• [ ] Access outputs of another terraform state (as done with data.terraform_remote_state) This is a problem because cross-team access to outputs require cross-team access to state which could be really risky given the secrets in the state. While we can use what I showed before, its not intuitive and straight forward. Simply splitting state and output would greatly improve the workflows across teams.

[bgshacklett]

@zerkms In that particular case, comparison of the current and previous values is impossible anyway, so what we're really looking for is interpolation of the value into another resource. I suggest that, on an opt-in basis, this functionality be removed for specific values, as it is 100% reliant on storing the value in the state file, which is the very thing we're looking to avoid in this issue. It's an "eat your cake and have it, too" situation.

Ideally, some middle ground could eventually be found, such as allowing a one-time interpolation of the value while the data remains in memory. This would allow for the creation of a secret in the user's preferred secrets management system which could be referenced at a later time, taking it out of Terraform's scope.

@daveadams You're suggesting reversible encryption via KMS or other mechanism? I don't believe this goes far enough. I will acknowledge that the granularity of access control would be much better, allowing for referencing of other teams' state files without being able to access their secrets. It would not prevent a user who has privileges to run Terraform from being able to retrieve application-level secrets after they're created, though. If Terraform has the capability to decrypt these secrets, then so does that user.

[zerkms]

In that particular case, comparison of the current and previous values is impossible anyway

@bgshacklett it's not needed: in that particular case you need to store the secret value into the state.

My point was that hashing is not generic and does not work.

which is the very thing we're looking to avoid in this issue.

we don't, the state should be encrypted then this is not an issue anymore.

[gtmtech]

Only having support for encrypting the whole statefile via a remote backend is not very good IMO. Firstly if you want encryption it forces you to use a remote backend that supports it, which is limiting the choice of users, (no i dont want to use terraform cloud) - and secondly its far too course-grained, not allowing scenarios such as Dave supplies above where different people have different concerns on a statefile.

Like a lot of terraform's "solutions" I find this one is geared to a not-very-big project where you dont have the kind of concerns that you do in large companies where granular security is important.

I had the same argument over at ansible with their next-to-useless ansible-vault which encrypts the entire vars file, not allowing meaningful diffs of changed elements. I had the same argument over in puppet years ago where their puppet-hiera-gpg encrypted an equivalent file and ended up joining forces with Tom Poulton to write puppet-hiera-eyaml which encrypts individual items instead. It wasnt perfect (using a single encryption key for starters), but people loved it at the time because it enabled diffs etc and could be built upon to provide multikey encryption). Hashicorp Vault is full of good ideas around encryption, I'm surprised they havent got more involved in helping terraform out, the integration between terraform and vault isn't as good as it needs to be (which is surprising for Hashicorp as a company, right?)

At least adding a pgp_key to an individual item on supported resources provided an attempted (but ugly) solution for not wanting to have sensitive values in the state whilst having different teams concerns addressed in the same statefile. Its a shame to see that being deprecated everywhere and not replaced with anything else. For me its the end user that should be able to decide if a particular item is sensitive or not, not because a developer has put a Sensitive flag in the code - and in the event the end user has decided its sensitive, they should be able to assert that (1) it should not be stored in the state at all, (2) should be stored but encrypted with a supplied key, (3) should be stored but encrypted via a pluggable API e.g. Vault. And finally (4) on top of all that the backend could provide additional encryption of the whole thing if necessary fulfilling the encryption-at-rest requirement if the user doesnt like other automatic encryption-at-rest that is given to them via e.g. s3 buckets.

People often confuse the many purposes of encryption - one concern (as in encrypting the whole thing) is for encryption-at-rest for compliance purposes, which is ONLY about people siphening information off of hard drives plucked from the computers they were running in. Its very important to have, but it does not tackle other purposes of encryption - encryption in transit, guarding information from people within your own and associated teams, in which case granular value-based encryption is also essential in a highly collaborative and large working environment which also needs to adhere to many internal informational standards such as ISO27001.

It would seem that terraform has done the first assuming it will solve the last. I'd like to see a multi-purpose approach which could cater for the others. I'm doing ridiculous workarounds such as setting a password to FOO with a dependent and triggered null_resource which resets it to BAR so BAR doesnt end up in the state. I'd like to see encryption as a first class supported candidate in terraform which has been well thought through rather than more workarounds.

[bernata]

The scenario for iam secret access key is in my view a compelling reason to support encrypting sensitive values. An example problematic scenario today is:

1. a dev in a team wants to run terraform to create production infrastructure
2. The terraform run outputs iam secret access key for an iam user created in the terraform.
3. The dev reads the credentials and writes them on the on premise server in the .aws/credentials file on an on premise server.
4. The tfstate is stored in an encrypted s3 bucket with the same secret access key.

The dev should not really have access to these production credentials. Creating a pipeline to run the terraform would help by limiting access to the tfstate and the outputted credentials; but is a fairly big infrastructure requirement. Encrypting the outputted credentials to a public key means the on-premise servers would decrypt and write the .aws/credentials at startup w/o humans being able to see the secret. It also means the credentials could be checked into source control [encrypted] making deployment of the credentials to on premise servers safe.

Please consider handling this specific scenario and supporting multiple encryption schemes and not just pgp; I created a PR 13119 to support RSA OAEP encryption of secret access keys in the aws provider.

[acdha]

bottom line, why aren't you encrypting the state file. A lot of comments just indicates that this is important.

Because it doesn’t solve the problems this issue was filed for. Encrypting at the file level means that everything using Terraform has to be trustworthy to handle all of your secrets. Storing secrets by reference or using field level encryption allows you to share state with tools and people who don’t require secrets to work.

As a simple example, think of analysis or reporting tools: currently you have to trust that tool’s authors and operators with all of your secrets even though they have no need for them.

Similarly, in a CI environment there’s no sure way to prevent someone from submitting a pull-request which leaks secrets during a pre-acceptance build. If those values aren’t in the state, you don’t have to worry about that particular risk because there’s no way to leak something which isn’t there.

[doribd]

agree - deleted, my mistake

[mr-miles]

I am encrypting the entire state file, but the problems arise when someone on the team needs to do a state move (which can be quite common) then they have to have access.

This must have been considered (maybe I have missed it in this thread) but what about just storing any variables marked as sensitive in a separate file, and having the normal state file contain placeholders for them?

Then the state file can be read freely, and the sensitive file can be further encrypted and have tight acls accordingly. When doing a state pull, there could be an option to replace sensitive variables (or something) so pull gives a workable state file but with the sensitive variables overwritten. Also maybe it wouldn't require changes to the individual providers and their change detection, since the "magic" would take place when terraform reads/writes the state.

[Plasma]

I’m new to terraform and found this surprising. Is it possible for me to flag a property a write-once so I can just remove it from all state/files entirely after initial creation?

thank you

[ausmith]

I’m new to terraform and found this surprising. Is it possible for me to flag a property a write-once so I can just remove it from all state/files entirely after initial creation?

~Yes, you can use the lifecycle.ignore_changes setting to specify what fields TF should ignore. https://www.terraform.io/docs/configuration/meta-arguments/lifecycle.html#ignore_changes -- Unfortunately, that doesn't solve the overall issue of having TF do something more with secrets than storing in plain text.~

As @cfbao points out in the following comment, my understanding of lifecycle.ignore_changes is inaccurate. It does put the contents of the data it "ignores" into the state file, just doesn't consider it for update operations.