search

hashicorp / vault-secrets-operator

The Vault Secrets Operator (VSO) allows Pods to consume Vault secrets natively from Kubernetes Secrets.
https://hashicorp.com
Other
471 stars 102 forks source link

AWS IAM Auth seems to ignore X-Vault-AWS-IAM-Server-ID header in VaultAuth CR #915

Open joshbench opened 2 months ago

joshbench commented 2 months ago

Describe the bug

When vault-secrets-operator is configured to authenticate to Vault with AWS IAM, with X-Vault-AWS-IAM-Server-ID set as a header in the VaultAuth custom resource, a VaultStaticSecret that is configured to use the configured VaultAuth fails to authenticate with the following error:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

It seems like when the VaultStaticSecret attempts to refresh, or to authenticate before refreshing, it is not attempting to use this header.

To Reproduce Steps to reproduce the behavior:

  1. Deploy application with the following yaml file with the following VSO custom resources.
helm install vault-secrets-operator hashicorp/vault-secrets-operator -n vault-secrets-operator --create-namespace -f values.yaml --version 0.8.1

defaultVaultConnection: enabled: true address: "https://my.vault.address:8200" tlsServerName: "my.vault.address" headers: X-Vault-AWS-IAM-Server-ID: "my.vault.address"

defaultAuthMethod: enabled: true method: aws mount: us-east-1/aws aws: role: my-role region: us-east-1 headers: X-Vault-AWS-IAM-Server-ID: "my.vault.address"

- example_secret.yml

apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: example-secret namespace: my-app-namespace spec: type: kv-v2 mount: secretmount path: my/secret/path destination: name: example-secret create: true overwrite: true refreshAfter: 1h

3. See error (vault-secrets-operator logs, application logs, etc.)

- `kubectl describe vaultstaticsecret example-secret`

Name: example-secret Namespace: my-app-namespace Labels: Annotations: API Version: secrets.hashicorp.com/v1beta1 Kind: VaultStaticSecret Metadata: Creation Timestamp: 2024-09-12T19:52:52Z Generation: 1 Resource Version: 56303 UID: c2470e10-ac7e-427e-a269-ffa2d031ba8f Spec: Destination: Create: true Name: example-secret Overwrite: true Hmac Secret Data: true Mount: secretmount Path: my/secret/path Refresh After: 1h Type: kv-v2 Events: Type Reason Age From Message

Warning VaultClientConfigError 4m6s (x203 over 19m) VaultStaticSecret Failed to get Vault auth login: Error making API request.

URL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login Code: 400. Errors:

Expected behavior

  1. VaultAuth is configured with the correct IAM server header
  2. A VaultStaticSecret is created
  3. The controller manager sees that a new secret is created
  4. The controller manager authenticates to Vault to fetch the secret. The controller manager uses all of the VaultAuth config, including any specified headers, to log in to Vault.
  5. The secret is pulled and stored in the specified location

Environment

Additional context Add any other context about the problem here.

benashz commented 2 months ago

Hi @joshbench,

Thanks reporting this issue. Would you mind setting the value on the VaultAuth's .spec.aws.headerValue, rather than in .spec.headers. See https://developer.hashicorp.com/vault/docs/platform/k8s/vso/api-reference#vaultauthconfigaws for more info.

Please let us know if that helps to resolve the issue.

Thanks,

Ben

joshbench commented 2 months ago

I @benashz,

Thank you for responding!

I reconfigured my values.yml like so:

---
controller:
  manager:
    extraEnv:
    # Required for Pod Identity (https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)
    - name: AWS_SDK_LOAD_CONFIG
      value: "'true'"

defaultVaultConnection:
  enabled: true
  address: "https://my.vault.address:8200"
  tlsServerName: "my.vault.address"

defaultAuthMethod:
  enabled: true
  method: aws
  mount: us-east-1/aws
  aws:
    role: my-role
    region: us-east-1
    headerValue: my.vault.address

And I'm still getting an error, but now it's "missing Authorization header":

{"level":"error","ts":"2024-09-13T18:27:09Z","logger":"cachingClientFactory","msg":"Failed to get NewClientWithLogin","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","VaultStaticSecret":{"name":"example-secret","namespace":"my-app-namespace"},"namespace":"my-app-namespace","name":"example-secret","reconcileID":"8b72bed4-4bd3-4a37-82f7-ae91f4b8995a","cacheKey":"aws-6bd141d8ace9805f592ec3","error":"Error making API request.\n\nURL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login\nCode: 400. Errors:\n\n* error validating X-Vault-AWS-IAM-Server-ID header: missing Authorization header"}

URL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login Code: 400. Errors:

URL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login Code: 400. Errors:

Seems like the Authorization header is possibly getting overwritten?

joshbench commented 2 months ago

Looking further into this, and also reading up on https://developer.hashicorp.com/vault/docs/auth/aws#iam-auth-method, I think I'm just not authenticating to AWS correctly, so the Authorization header is not being set. I think the version of the package used for AWS authentication uses an older version of the AWS SDK that doesn't support Pod Identity.

Where newer versions of awsutil seem to use aws-sdk-go-v2, which does support pod identity:

So maybe instead, this should be a feature request to support authentication via Pod Identity?