search

hashicorp / terraform-provider-vault

Terraform Vault provider
https://www.terraform.io/docs/providers/vault/
Mozilla Public License 2.0
467 stars 542 forks source link

Duplicated vault namespace - vault_auth_backend v3.16.0 #1903

Open arcezd opened 1 year ago

arcezd commented 1 year ago

Hi there,

We started facing an issue using the vault terraform provider 3.16.0, where the vault namespace seems to be duplicated at the vault API call, maybe it's related to concatenating the vault namespace used by the login token and the namespace specified at the vault resource.

Terraform Version

v1.5.0

Affected Resource(s)

Please list the resources as a list, for example:

This may be affecting other resources.

Terraform Configuration Files

variable "vault_namespace" {
  type        = string
  description = "Vault namespace"
}

variable "k8s_api_server" {
  type        = string
  description = "Kubernetes api server url"
}

variable "k8s_ca_cert" {
  type        = string
  description = "Kubernetes api server ca cert"
}

variable "k8s_vault_injector_token" {
  type        = string
  description = "Kubernetes vault injector token"
}

variable "k8s_issuer" {
  default     = "https://kubernetes.default.svc.cluster.local"
  type        = string
  description = "Kubernetes issuer"
}

variable "k8s_disable_iss_validation" {
  default     = "true"
  type        = string
  description = "Kubernetes disable issuer validation"
}

resource "vault_kubernetes_auth_backend_config" "k8s" {
  namespace = var.vault_namespace

  backend                = vault_auth_backend.k8s.path
  kubernetes_host        = var.k8s_api_server
  kubernetes_ca_cert     = var.k8s_ca_cert
  token_reviewer_jwt     = var.k8s_vault_injector_token
  issuer                 = var.k8s_issuer
  disable_iss_validation = var.k8s_disable_iss_validation
}

Debug Output

╷
│ Error: error reading from Vault: Error making API request.
│ 
│ Namespace: acc1/aws/namespace-01/acc1/aws/namespace-01
│ URL: GET https://vault-endpoint/v1/sys/auth
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│   * permission denied
│ 
│ 
│ 
│   with module.vault_k8s_auth.vault_auth_backend.k8s,
│   on ../../modules/vault-k8s-auth/main.tf line 1, in resource "vault_auth_backend" "k8s":
│    1: resource "vault_auth_backend" "k8s" {
│ 
╵

Panic Output

Expected Behavior

What should have happened? The GET https://vault-endpoint/v1/sys/auth call should succeed with the proper namespace: acc1/aws/namespace-01.

Actual Behavior

What actually happened? The GET https://vault-endpoint/v1/sys/auth call fails showing a duplicated namespace: acc1/aws/namespace-01/acc1/aws/namespace-01.

Steps to Reproduce

We're executing this using a GitHub actions workflow, but it seems like:

  1. Get the Vault token using the GitHub JWT. 
    github-workflow.yaml
- id: github-oidc-token
    name: Get GitHub OIDC Token
    uses: actions/github-script@v6
    with:
      script: |
        const coredemo = require('@actions/core')
        let id_token = await coredemo.getIDToken()
        coredemo.setOutput('gh_id_token', id_token)
  - id: vault-login
    name: Login to Vault
    run: |
      vault --version
      export VAULT_TOKEN=$(vault write -field=token ${{ vars.VAULT_AUTH_PATH }} role=${{ vars.VAULT_AUTH_ROLE }} jwt=${GITHUB_OIDC_JWT_TOKEN})
      vault login -no-store -field=policies $VAULT_TOKEN
      vault status
      echo "TOKEN=$(echo $VAULT_TOKEN)" >> $GITHUB_OUTPUT
    env:
      GITHUB_OIDC_JWT_TOKEN: ${{ steps.github-oidc-token.outputs.gh_id_token }}
      VAULT_NAMESPACE: ${{ vars.VAULT_NAMESPACE }}
  2. terraform plan

Important Factoids

Running in GitHub Workflow

References

VitaliyZasada commented 1 year ago

Confirmed. We started to receive permission denied error for vault_database_secret_backend_connection resource in 3.16.0 version, so downgraded to 3.15.2

image

Xavier59 commented 1 year ago

Yes, same problem here.

We have the following configuration:

provider "vault" {
  address = var.vault_addr
  auth_login {
    path      = "auth/approle/login"
    namespace = "admin"

    parameters = {
      role_id   = var.role_id
      secret_id = var.secret_id
    }
  }
}

Request should have header X-Vault-Namespace: admin but has header X-Vault-Namespace: admin/admin We decided to downgrade for now.

Could it be coming from https://github.com/hashicorp/terraform-provider-vault/pull/1841 ? @benashz

Xavier59 commented 1 year ago

Investigated a bit ... This is present in the logs:

[INFO] provider.terraform-provider-vault_v3.16.0_x5: 2023/06/21 14:58:46 [WARN] The provider namespace should be set whenever using namespaced auth tokens. You may want to update your provider configuration's namespace to be "admin", before executing terraform. Future releases may not support this type of configuration

So up to https://github.com/hashicorp/terraform-provider-vault/blob/31207b8ac3d7427add96d5ce50f11c3e8d6d966e/internal/provider/meta.go#L295-L311

The namespace seems to be correct (i.e not admin/admin). I'm not sure however why the warning pop up since the namespace is specified in my configuration.

So this my not work as expected ?

https://github.com/hashicorp/terraform-provider-vault/blob/31207b8ac3d7427add96d5ce50f11c3e8d6d966e/internal/provider/meta.go#L227

iniinikoski commented 1 year ago

Yes, same problem here.

We have the following configuration:

provider "vault" {
  address = var.vault_addr
  auth_login {
    path      = "auth/approle/login"
    namespace = "admin"

    parameters = {
      role_id   = var.role_id
      secret_id = var.secret_id
    }
  }
}

Request should have header X-Vault-Namespace: admin but has header X-Vault-Namespace: admin/admin We decided to downgrade for now.

Could it be coming from #1841 ? @benashz

Same issue here. Following-up, decided to stay in 3.15.0 for now.

Sartigan commented 1 year ago

While upgrading from 3.15.2 to 3.16.0... I suffer from a similar issue but not exactly in the same way.

Our app_role is located at the root level and we define the namespace in the provider configurations only (not in the terraform resources) And It looks like with the new version of the provider, it adds the namespace in the login request; probably due to the concatenation issue described by the others above. Removing the namespace from the configuration seems to resolve the invalid role ID issue... but that's not the intended way.

│ Error: Error making API request.
│ 
│ Namespace: mynamespace
│ URL: PUT https://vault-endpoint/v1/auth/approle/login
│ Code: 400. Errors:
│ 
│ * invalid role ID
│ 
│   with provider["registry.terraform.io/hashicorp/vault"].cluster,
│   on providers.tf line 73, in provider "vault":
│   73: provider "vault" {

With the following provider configurations:

provider "vault" {
  address = var.vault_endpoint
  namespace = var.vault_namespace
  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id   = var.approle_id
      secret_id = var.approle_secret
    }
  }
}

But by removing the namespace from the provider configurations and adding it to the terraform resources instead, the plan won't work as it will try to check in the root and not in the newly defined namespace in the resource which was previously in the provider configuration.

│ Error: error reading from Vault: Error making API request.
│ 
│ URL: GET https://my-vault.com/v1/sys/auth
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│   * permission denied
│ 
╵
╷
│ Error: error reading from Vault: Error making API request.
│ 
│ URL: GET https://my-vault.com/v1/sys/policies/acl/my-policy
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│   * permission denied
│

Something is definitely wrong with the namespace starting from 3.16.0 and most likely due to https://github.com/hashicorp/terraform-provider-vault/pull/1841

We can't upgrade, we have to stay on version 3.15.2 for now...

Callek commented 4 months ago

Casual bump to @benashz (who might pull in others) - This has blocked me updating for a while now, and I start to get concerned when we near a year with no plausible update paths for critical infra like this.

iniinikoski commented 4 months ago

Thanks for raising this @Callek. We'll also open a support ticket on this one soon if the comments stay quiet as this is a true blocker.

iniinikoski commented 2 months ago

Has anyone found any new information on this...? I bumped into this when rewriting our Vault modules (again) and this is somewhat very annoying. I'll probably raise this to Hashicorp's support next week (as we're are customers)... It's especially nasty when they fixed the lazy-loading of provider configs in version 3.23.0 but we cannot bump this provider version into production as it'd create the namespaces on root-level :/

iniinikoski commented 2 months ago

Replying to myself here: we've discovered the issue, probably.

Version 3.22.0 brough this option into the provider: https://github.com/hashicorp/terraform-provider-vault/blob/main/CHANGELOG.md#3220-nov-1-2023 - and - Hashicorp decided to make this to default to true. I believe this has broken many setups trying to use newer versions. We had to set this to false everywhere and then we were able to bump the provider to all the way to latest version 4 (4.4.0 if I remember correctly atm).

Maybe things got broken in https://github.com/hashicorp/terraform-provider-vault/blob/main/CHANGELOG.md#3210-oct-9-2023 when another bug was fixed (see "Fix failure when auth_login is specified and vault token is picked up from the runtime/execution environment"). Soo, you fix a bug and break things elsewhere...