Requesting temporary credentials in a way which allows Terraform to refresh them when they've expired

[rwblokzijl]

Terraform Version

Terraform v1.3.3
on linux_amd64

Use Cases

Some data sources like google_service_account_jwt. Generate a token that is valid for 1 hour.

However, since data sources only run at plan time. This means the JWT is sometimes expired by the time terraform apply is run. (we have an internal review process for terraform plans that sometimes takes longer than an hour).

FYI: The terraform provider for HashiCorps own Vault also requires a JWT for login with a google account.

Attempted Solutions

If we make the JWT dependent on something that only runs at apply time, it doesn't run at plan time anymore. Which is a problem since the JWT in an input for the Hashicorp Vault terraform provider.

Proposal

Add a flag that reruns the data source at apply. This allows the data to be fresh for the apply.

References

No response

[apparentlymart]

Hi @rwblokzijl! Thanks for sharing this feature request.

You have both shared a use-case (thanks!) and proposed a possible solution here.

I think there are potentially other ways to think about solving this problem. This is somewhat outside the intended scope for data sources, even though indeed a bunch of providers have chosen to use data sources for this situation due to it being the closest concept. But I think it's worth elevating the underlying problem because it might indicate that there's a missing concept in Terraform which is something separate from data sources:

Terraform runs sometimes depend on temporary external objects such as time-limited credentials, but currently Terraform has no real means to manage those temporary objects itself, and so there are two main workarounds for that missing feature today:

1. Use a data source and just hope that you run terraform apply soon enough after terraform plan that the object is still valid.
2. Manage the temporary object outside of Terraform and pass its values in to Terraform either as environment variables or Terraform input variables.

Both of these are problematic in that the external object can potentially expire before Terraform is done with its work, and Terraform can't refresh it because Terraform doesn't have any awareness of the expiration time or how to refresh it.

Extending the concept of data resources to represent something about expiration could be a viable solution, or we might find it better to introduce something entirely new that manages this problem.

I'm going to relabel this issue so that its title focuses on the problem rather than on a specific solution, because I think that will make it easier for others with similar situations to find it and vote on it (by adding :+1: reactions to the original comment) so we can weigh it against other work.

Thanks!

[apparentlymart]

Although it doesn't necessarily need to be part of the solution here, I want to note a related use-case that I've heard before:

When Terraform requests temporary credentials, it can be desirable for Terraform to also explicitly release those credentials once it has finished using them, to limit the validity period of those credentials as tightly as possible.

One advantage of first-class modelling of this situation, rather than just overloading data resources some more, would be the possibility of managing the "create" and "release/destroy" actions explicitly during every run.

[joe-a-t]

I just want to point out that in general using a Terraform data source to create a token/access key/other credential is generally not the best idea because the resulting credential is written to the Terraform statefile so anyone that has access to the statefile now has access to grab the creds and do the things. For the time being, I would strongly recommend handling the credential creation/management outside of terraform itself.

If Terraform were to introduce a new credentials first-class entity type or something instead of the current data source hack that some providers have created, that would be strongly preferred from my perspective since that new entity type could avoid writing the credentials to the statefile.

[sftim]

I agree with https://github.com/hashicorp/terraform/issues/32100#issuecomment-1293855182

Provider exec plugins for credentials are a good start, and are a third alternative to:

• Use a data source and just hope that you run terraform apply soon enough after terraform plan that the object is still valid.

• Manage the temporary object outside of Terraform and pass its values in to Terraform either as environment variables or Terraform input variables.

For example, hashicorp/kubernetes supports Kubernetes' exec credential plugin mechanism. The provider can call that plugin as often as it reasonable wants to, even if each token has quite a short lifespan.

(The downside: right now, you need to install and configure software locally that's separate from a Terraform provider.)

So, what I think we want is to strongly steer Terraform users to provider-level mechanisms for impersonating service accounts or acquiring special-user tokens or whatever, rather than the data source lookup approach. This also avoids storing credentials into state files and plan files.

[rwblokzijl]

We just ran into this again. We solved our initial problem with the TERRAFORM_VAULT_AUTH_JWT, which can be generated when we start an apply. But now we have the same problem with another credential we need for vault.

provider "vault" {
  address = var.vault_address
  auth_login_jwt {
    mount = "gcp"
    role  = var.vault_role
    jwt   = null # Set in env
  }
  headers {
    name  = "Authorization"
    value = "Bearer ${data.google_service_account_id_token.vault_iap_oidc.id_token}" # TODO: get OIDC from env
  }
}

However there is no env var supported in the vault provider for this purpose so there is no way for us to get this value at apply time.

Passing the token through terraforms variables is also not possible, since these are also only evaluated at plan time.

I think a general solution that terraform could greatly benefit from is to have some way to get data at apply time. Perhaps a special variable flag or data block that is "known on apply".

[apparentlymart]

Hi @rwblokzij! Thanks for sharing that use-case.

We'll include that in the mix when considering possible general designs here, but if it seems like an environment variable in the Vault provider would be a direct solution to your problem I would suggest also suggesting that in the hashicorp/vault provider's repository so the developers of that provider can see it and consider it.

Being able to set all "identity-related" provider settings via environment variables is typically a desirable goal anyway, regardless of it also serving as a way to deal with the use cases described here, so I expect the provider team would be open to it unless there's some special consideration for this particular setting that I'm not aware of.

Thanks again!