Helm chart v0.25.0 running on EKS 1.24 fails to assume IAM Role granting KMS key access

[jbouse]

Describe the bug Upgrading Helm chart from v0.24.1 to v0.25.0 on AWS EKS 1.24 cluster, upon restarting the new pod running 1.14.0 the container is unable to unseal the vault using AWS KMS key and log shows access denied to the KMS key with the assumed role of the EKS node not the one assigned to the service account via annotation.

I've observed the same behavior on multiple EKS clusters and reverting back to 0.24.1 chart works.

To Reproduce Steps to reproduce the behavior:

1. Install chart v0.24.1
2. Upgrade chart v0.25.0
3. Delete pod to restart with new chart
4. See error (vault logs, etc.)

❯ kubectl logs vault-2
Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/eks-workers-use1/i-07e1d1bfe3f196f93 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:us-east-1:XXXXXXXXXXXX:key/a5a4a1a8-XXXX-XXXX-XXXX-c41693e02423 because no identity-based policy allows the kms:DescribeKey action
    status code: 400, request id: cad0f64f-XXXX-XXXX-XXXX-cc46eaa2f6df
2023-07-05T16:20:40.518Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2023-07-05T16:20:40.518Z [WARN]  storage.raft.fsm: raft FSM db file has wider permissions than needed: needed=-rw------- existing=-rw-rw----

❯ kubectl describe statefulset vault
Name:               vault
Namespace:          vault
CreationTimestamp:  Tue, 08 Nov 2022 12:13:16 -0500
Selector:           app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault,component=server
Labels:             app.kubernetes.io/instance=vault
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=vault
                    k8slens-edit-resource-version=v1
Annotations:        meta.helm.sh/release-name: vault
                    meta.helm.sh/release-namespace: vault
Replicas:           3 desired | 3 total
Update Strategy:    OnDelete
Pods Status:        3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/instance=vault
                    app.kubernetes.io/name=vault
                    component=server
                    helm.sh/chart=vault-0.25.0
  Service Account:  vault
  Containers:
   vault:
    Image:       hashicorp/vault:1.14.0
    Ports:       8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
      [ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
      [ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
      [ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
      [ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    Readiness:  exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
    Environment:
      HOST_IP:                    (v1:status.hostIP)
      POD_IP:                     (v1:status.podIP)
      VAULT_K8S_POD_NAME:         (v1:metadata.name)
      VAULT_K8S_NAMESPACE:        (v1:metadata.namespace)
      VAULT_ADDR:                http://127.0.0.1:8200
      VAULT_API_ADDR:            http://$(POD_IP):8200
      SKIP_CHOWN:                true
      SKIP_SETCAP:               true
      HOSTNAME:                   (v1:metadata.name)
      VAULT_CLUSTER_ADDR:        https://$(HOSTNAME).vault-internal:8201
      HOME:                      /home/vault
      VAULT_SEAL_TYPE:           awskms
      VAULT_AWSKMS_SEAL_KEY_ID:  <set to the key 'KMS_KEY_ID' in secret 'vault-kms-key-id'>  Optional: false
    Mounts:
      /home/vault from home (rw)
      /vault/config from config (rw)
      /vault/data from data (rw)
  Volumes:
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
   home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Volume Claims:
  Name:          data
  StorageClass:
  Labels:        <none>
  Annotations:   <none>
  Capacity:      10Gi
  Access Modes:  [ReadWriteOnce]
Events:          <none>

❯ kubectl get statefulset vault -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: vault
    meta.helm.sh/release-namespace: vault
  creationTimestamp: "2022-11-08T17:13:16Z"
  generation: 10
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vault
    k8slens-edit-resource-version: v1
  name: vault
  namespace: vault
  resourceVersion: "147859478"
  uid: bf66fa2c-55b9-42ce-affc-c408d5b2d767
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: vault
      app.kubernetes.io/name: vault
      component: server
  serviceName: vault-internal
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: vault
        app.kubernetes.io/name: vault
        component: server
        helm.sh/chart: vault-0.25.0
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: vault
                app.kubernetes.io/name: vault
                component: server
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - "cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;\n[
          -n \"${HOST_IP}\" ] && sed -Ei \"s|HOST_IP|${HOST_IP?}|g\" /tmp/storageconfig.hcl;\n[
          -n \"${POD_IP}\" ] && sed -Ei \"s|POD_IP|${POD_IP?}|g\" /tmp/storageconfig.hcl;\n[
          -n \"${HOSTNAME}\" ] && sed -Ei \"s|HOSTNAME|${HOSTNAME?}|g\" /tmp/storageconfig.hcl;\n[
          -n \"${API_ADDR}\" ] && sed -Ei \"s|API_ADDR|${API_ADDR?}|g\" /tmp/storageconfig.hcl;\n[
          -n \"${TRANSIT_ADDR}\" ] && sed -Ei \"s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g\"
          /tmp/storageconfig.hcl;\n[ -n \"${RAFT_ADDR}\" ] && sed -Ei \"s|RAFT_ADDR|${RAFT_ADDR?}|g\"
          /tmp/storageconfig.hcl;\n/usr/local/bin/docker-entrypoint.sh vault server
          -config=/tmp/storageconfig.hcl \n"
        command:
        - /bin/sh
        - -ec
        env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: VAULT_K8S_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: VAULT_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: VAULT_ADDR
          value: http://127.0.0.1:8200
        - name: VAULT_API_ADDR
          value: http://$(POD_IP):8200
        - name: SKIP_CHOWN
          value: "true"
        - name: SKIP_SETCAP
          value: "true"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: VAULT_CLUSTER_ADDR
          value: https://$(HOSTNAME).vault-internal:8201
        - name: HOME
          value: /home/vault
        - name: VAULT_SEAL_TYPE
          value: awskms
        - name: VAULT_AWSKMS_SEAL_KEY_ID
          valueFrom:
            secretKeyRef:
              key: KMS_KEY_ID
              name: vault-kms-key-id
        image: hashicorp/vault:1.14.0
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - sleep 5 && kill -SIGTERM $(pidof vault)
        name: vault
        ports:
        - containerPort: 8200
          name: http
          protocol: TCP
        - containerPort: 8201
          name: https-internal
          protocol: TCP
        - containerPort: 8202
          name: http-rep
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -ec
            - vault status -tls-skip-verify
          failureThreshold: 2
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /vault/data
          name: data
        - mountPath: /vault/config
          name: config
        - mountPath: /home/vault
          name: home
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 100
      serviceAccount: vault
      serviceAccountName: vault
      terminationGracePeriodSeconds: 10
      volumes:
      - configMap:
          defaultMode: 420
          name: vault-config
        name: config
      - emptyDir: {}
        name: home
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 2
  collisionCount: 0
  currentRevision: vault-5dfc46c856
  observedGeneration: 10
  readyReplicas: 2
  replicas: 3
  updateRevision: vault-5df4f9f6c8
  updatedReplicas: 1

Expected behavior Expected that restarting the pod would come back online successfully as every other upgrade has in the past.

Environment

• Kubernetes version: 1.24
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
• vault-helm version: 0.25.0

Chart values:

injector:
  nodeSelector:
    kubernetes.io/os: linux
server:
  extraEnvironmentVars:
    VAULT_SEAL_TYPE: awskms
  extraSecretEnvironmentVars:
  - envName: VAULT_AWSKMS_SEAL_KEY_ID
    secretKey: KMS_KEY_ID
    secretName: vault-kms-key-id
  ha:
    enabled: true
    raft:
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          x_forwarded_for_authorized_addrs = ["10.20.0.0/16"]
        }

        storage "raft" {
          path = "/vault/data"
        }

        telemetry {
          disable_hostname = true
          prometheus_retention_time = "12h"
        }

        service_registration "kubernetes" {}
      enabled: true
  ingress:
    annotations:
      alb.ingress.kubernetes.io/group.name: devops
      alb.ingress.kubernetes.io/group.order: "200"
      alb.ingress.kubernetes.io/healthcheck-path: /v1/sys/health?standbyok=true&perfstandbyok=true&sealedcode=204&uninitcode=204
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
      alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.x_amzn_tls_version_and_cipher_suite.enabled=true
      alb.ingress.kubernetes.io/scheme: internal
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-Res-2020-10
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/success-codes: 200,204
      alb.ingress.kubernetes.io/target-type: ip
    enabled: true
    hosts:
    - host: vault.example.com
    - host: vault-use1.example.com
    ingressClassName: alb
  nodeSelector:
    kubernetes.io/os: linux
  service:
    type: NodePort
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/vault-use1
    create: true
    name: vault

[cayla]

I experienced this on EKS 1.27, so it might not be version specific (or tightly constrained anyway).

What was really odd to me if that there really were not any significant differences between the stateful sets generated by 0.24.1 and 0.25. (Sorry all I have is a screenshot of it, not the actual text)

[jbouse]

Yes, I also compared 0.24.1 and 0.25.0 and couldn't see anything drastic changed that would have caused this behavior. Still, on multiple EKS clusters, I observed the same behavior, and reverting cleared up the issue, so something less obvious has changed.

It then begs the question, is the change in the chart or vault 1.14.0 itself? Perhaps I'll try setting server.image.tag to 1.14.0 on the 0.24.1 chart and see if I get the same failure. That would help narrow it down to the chart or Vault at fault.

[tomhjp]

I haven't looked into this yet, but I think you're onto the right track with it being 1.14.0 based on https://github.com/hashicorp/vault/issues/21465. This will be a pretty high priority to get fixed, I'd recommend following along in that issue for any updates.

[jbouse]

@tomhjp you appear correct there... I just finished running with server.image.tag: 1.14.0 on chart v0.24.1 and it failed the same way I was seeing with chart 0.25.0. So it does appear to be related to the vault ticket against 1.14.0. Reading the comments in that ticket it seems to be recommending the fix to include the role_arn and web_identity_token_file which seems to indicate that vault 1.14.0 is failing to read the environment variables applied to the pod as both AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are set along with other env variables the SDK should pick up and use.

[jbouse]

Okay, so I'm convinced this is a vault 1.14.0 issue, not the chart at fault here... I've upgraded to the 0.25.0 chart but set server.image.tag: 1.13.4, and it works perfectly fine. I'll maintain this until vault is fixed... I also found hashicorp/vault#21478, which describes what I've seen and includes a pointer to what broke.

I'll go ahead and close this, as it isn't the chart at fault.

[brunooon]

@jbouse I tried setting the server.image.tag: 1.14.1 and its also working now

[adrienbroyere]

Can confirm it works as expected with the Chart's version 0.25.0 using the 1.14.1 image. I was also faced with the issue of my IRSA not being properly used with the 1.14.0image tag.

tanks @brunooon 👍