SchemaStateFunc not being applied to Computed attributes

[alexng-canuck]

Terraform Version

terraform v0.11.8

Expected Behavior

We created a resource and datasource that has a Computed attribute and assigned it a SchemaStateFunc with following behavior, for example:

     "plaintext": {
                Type:     schema.TypeString,
                StateFunc: func(v interface{}) string {
                    log.Printf("trying to obfuscate plaintext")
                    return "<REDACTED>"
                },
                Computed: true,
            },

This should result in the obfuscated "REDACTED" value being stored in the state file.

Actual Behavior

The original value of the attribute is stored in state file.

Additional Context

Some data sources may return sensitive information (such as passwords or decrypted data) that is useful as an "output" value to be consumed by the user but should not be persisted in state.

Was hoping that the SchemaStateFunc could be used to generate some sort of hash or other override value to avoid storing the original sensitive information in state.

Marking the attribute as Sensitive is insufficient because we actually want the opposite: the value should be visible through output but not visible in state.

[apparentlymart]

Hi @alexng-canuck! Thanks for reporting this.

I've marked this as an SDK bug representing the fact that apparently the StateFunc isn't working in all cases, and that inconsistency is not intended.

However, I wanted also to note that the use-case you mentioned wouldn't be addressed by this change, because expression references return the value from the state: it'll either be redacted both in the state and in derived values, or neither. The StateFunc mechanism was originally intended to allow storing a more compact version of the value, e.g. a minified version of a JSON value: it's not intended to be used to obfuscate sensitive data. As we currently stand, the data is either included in the state or it isn't; unless this data needs to be passed somewhere else in the configuration, I'd suggest not exporting it as an attribute at all and having users access the sensitive value out-of-band somehow instead.

[alexng-canuck]

Thanks. In our case, we do want the user to be able to pass this sensitive attribute to another resource, but without storing it in plaintext in the state. (Almost like a passthrough).

Say the attribute is being referenced in another resource like this:

data supersensitiveresource foo {
   // has sensitive/computed attribute called "plaintext"
}

resource instance bar {
    password = "${foo.plaintext}"
}

Ideally, password would use the original foo.plaintext value at create time, while the StateFunc value is what's stored in state. Wondering if such a use case would be possible after your proposed fix.

More generally speaking, we want users to be able to directly supply passwords and other sensitive fields for provisioning resources without it ever showing up in the state file in plaintext.

[apparentlymart]

At this time, there is no way for an attribute value to be referenceable in expressions without it being stored in the state, and there's a key problem with any such mechanism: on a subsequent run Terraform needs to be able to re-evaluate the expressions and get identical results in order to prove that no changes are required, and thus any values used in expressions must be persisted to be available on the next run.

The more likely future direction (though no specific design has been devised yet) is to continue to keep the values in the state but to have a mechanism to split the state into sensitive and non-sensitive separate parts, though there are still several fundamental problems to address with that model, such as: can Terraform actually do anything useful if you don't have access to the sensitive part, and if not does this have any value vs. just treating the whole state as a sensitive artifact? That's just one example; there are many more that I don't have on the top of my head right now.

hashicorp/terraform#516 is the umbrella tracking issue for this problem. Currently any implementation work is blocked on figuring out a design that meets all the needs without breaking Terraform's fundamental assumptions. In the mean time, Terraform CLI and providers must operate under the assumption that all attributes get stored in the state and respond accordingly, such as not exporting the data in the first place (and documenting some out-of-band mechanism for dealing with it) or by exporting it but including a prominent warning in the documentation (as we see in the Vault provider).

[lyda]

Either the docs should change or the behaviour should.

Having a value I can pass around while terraform is running but does not end up on disk when terraform is not running would be better. In my case I have a value I can get from a remote resource with the id and that i want to pass around to other resources, but I would prefer to just store a hash of it on disk.

I know terraform state files have secrets in them, but I don't see why we have to stuff them full to the brim with secrets.

[paddycarver]

We have added some guidance to the website on this issue.