Usage of `vault_terraform_cloud_secret_creds` does not work directly with TFE provider due to dependency issues

[mechastorm]

It seems that the TFE token generated from resource vault_terraform_cloud_secret_creds cannot be used in the TFE provider.

The root cause is most likely due to the fact that the TFE provider is not waiting for the token generated by vault_terraform_cloud_secret_creds.

The ideal fix would be if Terraform supported depends_on in the provider level but that is still an ongoing discussion - https://github.com/hashicorp/terraform/issues/2430

It feels like the resource vault_terraform_cloud_secret_creds should be instead a data resources perhaps (ie. data. vault_aws_access_credentials)? Though I'm curious to know what other suggestions that the terraform-provider-vault team can provide.

Terraform Version

Terraform v1.0.3
on darwin_amd64
+ provider registry.terraform.io/hashicorp/tfe v0.26.1
+ provider registry.terraform.io/hashicorp/vault v2.21.0

Affected Resource(s)

vault_terraform_cloud_secret_creds

Terraform Configuration Files

provider "vault" {
  address    = var.vault_address
}

resource "vault_terraform_cloud_secret_creds" "tfe" {
  backend = var.vault_tfe_backend_name
  role    = var.vault_tfe_backend_role_name
}

provider "tfe" {
  hostname = var.tfe_hostname
  token    = vault_terraform_cloud_secret_creds.tfe.token
}

resource "tfe_workspace" "test" {
  name         = "test-my-workspace-name"
  organization = var.tfe_org_name
}

Expected Behavior

Terraform Enterprise is able to create a TFE workspace using a token generated from Vault's TFE secret engine.

Actual Behavior

Unable to authenticate TFE provider with TFE token generated from Vault TFE secret engine

[ausfestivus]

I just hit this with a customer environment. In our case, while building out the Terraform code I had initially configured the TFE Provider using a simple

provider "tfe" {
  token = var.tfc_token
}

config.

Later on I switched over to using

provider "tfe" {
  token = vault_terraform_cloud_secret_creds.tfc.token
}

resource "vault_terraform_cloud_secret_creds" "tfc" {
  backend = local.vault_tfc_token[var.environment].backend
  role    = local.vault_tfc_token[var.environment].role
}

And the Workspace continued to operate as normal despite the Vault resource config being in place with the TFE Provider.

Following on from that a week later, I deleted and recreated the Workspace in TFC. The first plan in the recreated Workspace returned

Error: Error retrieving provider meta values for internal provider.
│ 
│   with provider["registry.terraform.io/hashicorp/tfe"],
│   on main.tf line 43, in provider "tfe":
│   43: provider "tfe" {
│ 
│ This should never happen; please report it to
│ https://github.com/hashicorp/terraform-provider-tfe/issues
│ 
│ The error received was: "Could not set the token value to string
│ unmarshaling unknown values is not supported"
╵
Operation failed: failed running terraform plan (exit 1)

So, it looks like the TFE Provider didnt pick up the change in its config either. I'll raise an issue over in the TFE Provider repo momentarily and link it here.

[ausfestivus]

Workaround:

provider "tfe" {
  token = data.vault_generic_secret.tfc.data["token"]
}

locals {
  vault_tfc_token = {
    dev = {
      backend = "acl_cpt/tfc_dev"
      role    = "tfc-orgname-dev-org-token"
    }
    prod = {
      backend = "acl_cpt/tfc_prod"
      role    = "tfc-orgname-prod-org-token"
    }
  }
}

data "vault_generic_secret" "tfc" {
  path = "${local.vault_tfc_token[var.environment].backend}/creds/${local.vault_tfc_token[var.environment].role}"
}