.token_reviewer_jwt: inconsistent values for sensitive attribute.

[chap-dr]

Terraform Version

1.0.2

Affected Resource(s)

• vault_kubernetes_auth_backend_config

Terraform Configuration Files

data "kubernetes_service_account" "sa" {
  depends_on = [module.vault_secret_webhook]

  metadata {
    name      = local.base_name
    namespace = local.destination_namespace
  }
}

data "kubernetes_secret" "sa_secret" {
  depends_on = [module.vault_secret_webhook]

  metadata {
    name           = data.kubernetes_service_account.sa.default_secret_name
    namespace = local.destination_namespace
  }
  binary_data = {
    "ca.crt" = ""
    token    = ""
  }
}

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "${var.var1}_${var.var2}"
  tune {
    default_lease_ttl = "24h"
  }
}

resource "vault_kubernetes_auth_backend_config" "ka" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://${var.cluster_endpoint}"
  kubernetes_ca_cert     = base64decode(data.kubernetes_secret.sa_secret.binary_data["ca.crt"])
  token_reviewer_jwt     = base64decode(data.kubernetes_secret.sa_secret.binary_data["token"])
  disable_iss_validation = "true"
}

Expected Behavior

tf apply without error

Actual Behavior

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for
│ module.vault-auth-backend[0].vault_kubernetes_auth_backend_config.ka to
│ include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/vault" produced an invalid new value for
│ .token_reviewer_jwt: inconsistent values for sensitive attribute.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵

Error: Apply operation failed

Important Factoids

Vault provider v. 3.6.0 Vault v. 1.10

This happens both when running terraform cli local and in a remote runner.

I've tried adding sleep timers between sa creation and data source load, but no matter how high I went, I kept getting the same error.

Moving vault_auth_backend and vault_kubernetes_auth_backend_config and sending output of the data sources from one module to another mitigated the issue somewhat, it stopped it from happening consistently, and made it happen perhaps 1 out of 10 runs.

[benashz]

I am not sure this is related to this issue but I think that that the values for kubernetes_ca_cert and token_reviewer_jwt may be reversed in your example resource.

[chap-dr]

I am not sure this is related to this issue but I think that that the values for kubernetes_ca_cert and token_reviewer_jwt may be reversed in your example resource.

Right.. that's a copy paste error on my part when re doing the tf code that errors consistently, I'll fix that right away, I have this working, by splitting it over two modules, however as i mentioned, we do still see the error occasionally with the two module workaround.

Apologizes for the confusion, and thanks for the heads up and quick catch!

[benashz]

Ah, no worries. Thanks for the clarification! I do see a potential culprit in the code. Going to do some preliminary testing to confirm.

[benashz]

I wonder if you wouldn't mind describing your setup/workflow a bit?

For example:

• How many modules do you have for each vault provider{} block?
• How many Vault server instances are being provisioned per terraform apply?
• Does each module correspond to a distinct vault_auth_backend path?

Thanks!

[chap-dr]

How many modules do you have for each vault provider{} block?

Not sure i fully understand? We have one vault provider block in providers.tf, and a total of 12 modules in main.tf(this number will grow as we further develop the project) these modules to many things, alot of which are not related to Vault at all, We load data from Vault a couple of places but only write an auth backend in one module.

How many Vault server instances are being provisioned per terraform apply?

None 😄, This project is for bootstrapping Kubernetes clusters, so we have one terraform project that provisions infrastructure (the GKE cluster other infra needed) and then this project which bootstraps the cluster with the "core-services" we need to run applications, i.e external-dns, cert-manager and the likes.

Does each module correspond to a distinct vault_auth_backend path? The one module, creating a kubernetes auth backend is generating its path name based on two set values.

Hope that answers your questions.

Thanks.

[chap-dr]

@benashz Any ideas? :)

[klakabaka]

We have the same issue in the same case! Could it be fixed?

Thanks

[kalenarndt]

I was able to get around this recently by referencing the raw values on the secret instead of looking up the base64 encoded value and then decoding it on the auth backend config

data "kubernetes_service_account" "test" {
  metadata {
    name = var.kubernetes_vault_service_account
    namespace = data.kubernetes_namespace.vault.id
  }
}

# Retrieve the token and ca cert used for Vault k8s authentication
data "kubernetes_secret" "vault" {
  metadata {
    name      = data.kubernetes_service_account.test.default_secret_name
    namespace = data.kubernetes_namespace.vault.id
  }
}

resource "vault_auth_backend" "kubernetes" {
  path = var.vault_auth_path
  type = var.vault_auth_type
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://10.43.0.1"
  kubernetes_ca_cert     = data.kubernetes_secret.vault.data["ca.crt"]
  token_reviewer_jwt     = data.kubernetes_secret.vault.data.token
  disable_local_ca_jwt   = true
  disable_iss_validation = true
}`

[klakabaka]

@kalenarndt Great, thanks! Finally solved the problem 👍