search

hashicorp / vault-k8s

First-class support for Vault and Kubernetes.
Mozilla Public License 2.0
790 stars 171 forks source link

Azure auth failure with user assigned Identity #396

Open masterphenix opened 2 years ago

masterphenix commented 2 years ago

Describe the bug I am getting the following errors in the injected init container when trying to use Azure Auth with the Vault Injector, and a user assigned Identity assigned to my pod. First a couple of these:

2022-10-21T14:33:06.097Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT http://vault.vault.svc:8200/v1/auth/azure/login
  | Code: 500. Errors:
  |
  | * token object id does not match virtual machine identities
   backoff=1s

And then, a whole lot of these until the pod is killed:

2022-10-21T14:33:30.086Z [ERROR] auth.handler: error getting path or data from method:
  error=
  | error response in metadata from http://169.254.169.254/metadata/identity/oauth2/token: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=<my-uai-client-id>&resource=https%3A%2F%2Fmanagement.azure.com%2F

To Reproduce Steps to reproduce the behavior:

  1. Prerequisites: have AAD Pod Identity Deployed, and Azure Auth configured and working on Vault
  2. Run an ubuntu pod with the following labels & annotations: 
    kubectl run ubuntu-debug --rm -i --tty --image ubuntu  --labels="aadpodidbinding=mypodid-binding" --annotations="vault.hashicorp.com/agent-inject=true" --annotations="vault.hashicorp.com/role=poc-uai-sandbox" --annotations="vault.hashicorp.com/agent-inject-secret-config.txt=kv/data/azure/app/poc-sandbox" --annotations="vault.hashicorp.com/agent-pre-populate=true" --annotations="vault.hashicorp.com/log-level=trace" --annotations="vault.hashicorp.com/auth-type=azure" --annotations="vault.hashicorp.com/auth-config-resource=https://management.azure.com/" -- bash
  3. See errors in the vault-agent-init init container of the pod: 
    2022-10-21T14:33:05.665Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"TRACE","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"{{ with secret \"kv/data/azure/app/poc-sandbox\" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/config.txt","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://vault.vault.svc:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000}
2022-10-21T14:33:05.665Z [INFO] (runner) creating watcher
2022-10-21T14:33:06.097Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://vault.vault.svc:8200/v1/auth/azure/login
| Code: 500. Errors:
|
| * token object id does not match virtual machine identities
backoff=1s
2022-10-21T14:33:07.098Z [INFO]  auth.handler: authenticating
2022-10-21T14:33:07.098Z [TRACE] auth.azure: beginning authentication
2022-10-21T14:33:07.245Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://vault.vault.svc:8200/v1/auth/azure/login
| Code: 500. Errors:
|
| * token object id does not match virtual machine identities
backoff=1.85s
2022-10-21T14:33:09.103Z [INFO]  auth.handler: authenticating
2022-10-21T14:33:09.103Z [TRACE] auth.azure: beginning authentication
2022-10-21T14:33:09.236Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://vault.vault.svc:8200/v1/auth/azure/login
| Code: 500. Errors:
|
| * token object id does not match virtual machine identities
backoff=3.43s
2022-10-21T14:33:12.669Z [INFO]  auth.handler: authenticating
2022-10-21T14:33:12.669Z [TRACE] auth.azure: beginning authentication
2022-10-21T14:33:12.700Z [ERROR] auth.handler: error getting path or data from method:
error=
| error response in metadata from http://169.254.169.254/metadata/identity/oauth2/token: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=<my-uai-client-id>&resource=https%3A%2F%2Fmanagement.azure.com%2F
backoff=6.39s

Application deployment:

# Init Container injected by vault injector looks like this (I have decoded the base64 config for more clarity):
  initContainers:
  - args:
    - echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    command:
    - /bin/sh
    - -ec
    env:
    - name: VAULT_LOG_LEVEL
      value: trace
    - name: VAULT_LOG_FORMAT
      value: standard
    - name: VAULT_CONFIG
      value: {"auto_auth":{"method":{"type":"azure","mount_path":"auth/azure","config":{"resource":"https://management.azure.com/","role":"poc-uai-sandbox","token_path":"/var/run/secrets/kubernetes.io/serviceaccount/token"}},"sink":[{"type":"file","config":{"path":"/home/vault/.vault-token"}}]},"exit_after_auth":true,"pid_file":"/home/vault/.pid","vault":{"address":"http://vault.vault.svc:8200"},"template":[{"destination":"/vault/secrets/config.txt","contents":"{{ with secret \"kv/data/azure/app/poc-sandbox\" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","left_delimiter":"{{","right_delimiter":"}}"}],"template_config":{"exit_on_retry_failure":true}}
    image: hashicorp/vault:1.11.3
    imagePullPolicy: IfNotPresent
    name: vault-agent-init
    resources:
      limits:
        cpu: 500m
        ephemeral-storage: 128Mi
        memory: 128Mi
      requests:
        cpu: 250m
        ephemeral-storage: 64Mi
        memory: 64Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 100
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/vault
      name: home-init
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-7d54p
      readOnly: true
    - mountPath: /vault/secrets
      name: vault-secrets

When the init-container is not injected, and the IMDS manually queried, a token is successfully acquired:

# kubectl run ubuntu-debug --rm -i --tty --image ubuntu  --labels="aadpodidbinding=mypodid-binding" -- bash
root@ubuntu-debug:/# curl -s 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=<my-app-client-id>&resource=https%3A%2F%2Fmanagement.azure.com%2F' -H Metadata:true
{"access_token":"xxxxxxxxxxxx","refresh_token":"","expires_in":"86400","expires_on":"1666449152","not_before":"1666362452","resource":"https://management.azure.com/","token_type":"Bearer"}

And, If I query vault manually from this point on with vault CLI, I can successfully access my secret by means of the user assigned Identity.

Expected behavior Init-container should be able to successfully use pod identity to query secrets.

Environment

Additional context N/A

kchervonets commented 1 year ago

@masterphenix did you find any solution? I have the same error with azure federation token now.

masterphenix commented 1 year ago

Hello, not sure it is considered a solution, but since AAD Pod Identity is now deprecated, we have moved to Azure Workload Identity, and it works fine.

kchervonets commented 1 year ago

@masterphenix Hi, for me Azure Workload Identity does not work, I have the same error

2023-11-20T21:43:59.828Z [ERROR] agent.auth.handler: error getting path or data from method: error="error response in metadata from http://169.254.169.254/metadata/identity/oauth2/token: {\"error\":\"invalid_request\",\"error_description\":\"Identity not found\"}" backoff=4m48.84s

Could you please provide a bit more details about your set up? Such as VAULT_CONFIG for example, so I can compare, thanks.

aleksandrovpa commented 8 months ago

@kchervonets Hi, I'm facing the same issues and I'm not sure what I'm doing wrong. Did you manage to get it working? I guess the problem is with HashiCorp Vault, not with the user-assigned identity

kchervonets commented 8 months ago

@aleksandrovpa I've never managed to make it worked with user assigned identities. It works fine with workload assigned identities, but user ones don't. I looked through vault injector's code and Azure library and didn't find any confirmation that it should work. I ended up using neither and used AKS as a oidc issuer and a auth/jwt method for authentication.

aleksandrovpa commented 8 months ago

@kchervonets could you please share the links about workload assigned identities? There are too many documents, and I have no idea which one works. I'd greatly appreciate it!