terraform-provider-tfe unable to find credentials

[chris3ware]

Hi, I wasn't sure whether to raise this here, or on the provider itself. 111 and 102 have discussed credential helpers and it does look like they are supported.

I have stored my terraform cloud user API key in my macOS key chain and configured my ~/.terraformrc file as follows:

➜  cat -p ~/.terraformrc

credentials_helper "keychain" {
  args = ["--keychain=terraform"]
}

I am using terraform cloud to store state in several workspaces. When I run terraform plan I can see output for resources from other providers refreshing OK and the debug log output shows that the API key for the remote state is being read using the credential_helper:

2022-09-28T16:06:43.232+0100 [DEBUG] Attempting to open CLI config file: /Users/chris/.terraformrc
2022-09-28T16:06:43.232+0100 [INFO]  Loading CLI configuration from /Users/chris/.terraformrc
2022-09-28T16:06:43.235+0100 [DEBUG] checking for credentials in "/Users/chris/.terraform.d/plugins"
2022-09-28T16:06:43.235+0100 [WARN]  found legacy credentials "terraform-credentials-keychain"

However, the plan fails when it comes to refreshing the state of the tfe provider resources:

2022-09-28T16:06:48.606+0100 [DEBUG] provider.terraform-provider-tfe_v0.36.1_x5: [DEBUG] Attempting to fetch token from Terraform CLI configuration for hostname app.terraform.io...
2022-09-28T16:06:48.606+0100 [ERROR] vertex "provider[\"registry.terraform.io/hashicorp/tfe\"]" error: Required token could not be found. Please set the token using an input variable in the provider configuration block or by using the TFE_TOKEN environment variable.

The tfe provider does work if the API key is set in the provider configuration, as an environment variable or in the ~/.terraformrc file.

Thanks.

[bendrucker]

Can you confirm that this is a local workspace, not a remote one (Terraform Cloud)?

[chris3ware]

This is a remote workspace.

[bendrucker]

This helper will only be used by the provider for workspaces in local execution mode.

https://developer.hashicorp.com/terraform/cloud-docs/run/remote-operations

When using full remote operations, operations like terraform plan or terraform apply can be executed in Terraform Cloud's run environment, with log output streaming to the local terminal.

The credential helper is being used to obtain a TF Cloud token so that configuration can be uploaded, a run can be executed, and logs can be streamed back. This prevents you from needing a static token on your local machine.

The tfe provider has never attempted to call the credential helper in your example. As the logs you posted indicate, provider logs are prefixed with the provider name/version.

The actual operations of the Terraform run, including refreshing, are performed remotely.

https://developer.hashicorp.com/terraform/language/settings/backends/remote

Remote plans and applies use variable values from the associated Terraform Cloud workspace.

The main Terraform process, as well as all the provider plugins, are running in a TFC environment. Local content, including files and environment variables, will not be accessible. Only explicitly passed variables, set either at the workspace (direct or variable set) or run level, are available. Configuration must not be specific to your local machine. Similarly, file(pathexpand("~/.ssh/id_rsa")) would fail in TFC if you expected it to load your SSH key from your local home directory.

This isn't really an issue to raise on the provider, but rather something that just isn't logically possible with remote Terraform execution. This can only be addressed at the Terraform Cloud level, by enabling the use of short-lived tokens to call its API. In other words, it shouldn't be transporting your local token into the cloud, it should inject a short-lived token into the cloud environment. TFC already issues tokens for each run, but only with minimal permissions needed to execute and report on the run, and no way to customize the token.

There are plenty of folks complaining about this, but seemingly unappreciative of the complexity of the problem:

https://github.com/hashicorp/terraform-provider-tfe/issues/102#issuecomment-606819967

It's certainly not an option for TFC to be open by default and allow any workspace to modify TFC config, even if that config is scoped to the workspace itself. Any authorization would have to be explicitly granted. Doable, surely, but not at all simple to implement within the existing authn/authz system in TF Cloud.

[chris3ware]

Hi, apologies for any confusion. When I said the workspace is "remote" - I meant that the workspace is on terraform cloud i.e. the state file is stored on terraform cloud. The execution mode for the workspace is local.

As i said in my initial comment

"The tfe provider does work if the API key is set in the provider configuration, as an environment variable or in the ~/.terraformrc file"

If, as you say, "The tfe provider has never attempted to call the credential helper in your example" - then perhaps an issue with the provider?

[bendrucker]

Ah ok. In that case it's effectively not a remote workspace. Local execution means TFC is treated as a generic storage/locking backend.

My comment about the credential helper getting invoked to access the backend and any private modules still holds. The logs you shared cover that, and not execution by the provider.

However, it's still possible the provider is calling the helper, that's just not proven by any of the logs.

I'll have to try this out and see what the actual behavior is.

[chris3ware]

Hi, I've had some feedback from the TFE provider issue: 637. It sounds like this may never work with the Terraform cloud provider.

[bendrucker]

See https://github.com/hashicorp/terraform-provider-tfe/pull/658. This is totally doable and they might accept a PR. Otherwise, given the limited usefulness, it's fair to say credential helper support is highly unlikely.

In cases where I need an env var instead (e.g. making API calls via curl) I'll often do this:

export TFE_TOKEN=$(/path/to/helper get app.terraform.io | jq -r .token)

[chris3ware]

Thanks for the tip. I also found a VSCode extension that handles sensitive environment variables. https://marketplace.visualstudio.com/items?itemName=pomdtr.secrets&ssr=false#overview