Helm chart with externalVaultAddr doesn't create "vault" service account which has "system:auth-delegator" permission

We are trying to install vault-agent in k8s clusters with external vault configuration. We enabled kubernetes auth and config with the generated vault agent service account and during runtime when we try to retrieve the secrets it throws the error that vault agent service account doesn't have enough permission

2023-06-02T23:07:58.981Z [ERROR] auth.kubernetes.auth_kubernetes_5fd71264: login unauthorized due to: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vault-example:vault-agent-injector\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope","reason":"Forbidden","details":{"group":"authentication.k8s.io","kind":"tokenreviews"},"code":403}

Upon checking out and cross verifying with vault documentation, it does expect "vault" service account with system:auth-delegator permission to be created along with vault-agent service account. I cross verified with dry-run command and it does NOT include vault service account when we enable externalVaultAddr param in Global.

Here is my command:

 (⎈ |bge-stg1:vault-example)@~ % helm install vault hashicorp/vault --set "global.externalVaultAddr=http://10.XXX.YY.ZZZ:31763" --version 0.23.0 --dry-run
NAME: vault
LAST DEPLOYED: Mon Jun  5 11:29:35 2023
NAMESPACE: vault-example
STATUS: pending-install
REVISION: 1
TEST SUITE: None
HOOKS:
MANIFEST:
---
# Source: vault/templates/injector-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-agent-injector
  namespace: vault-example
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/server-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault
  namespace: vault-example
  labels:
    helm.sh/chart: vault-0.23.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-agent-injector-clusterrole
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
rules:
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations"]
  verbs:
    - "get"
    - "list"
    - "watch"
    - "patch"
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-agent-injector-binding
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
  name: vault-agent-injector
  namespace: vault-example
---
# Source: vault/templates/server-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-server-binding
  labels:
    helm.sh/chart: vault-0.23.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault-example
---
# Source: vault/templates/injector-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-agent-injector-svc
  namespace: vault-example
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm

spec:
  ports:
  - name: https
    port: 443
    targetPort: 8080
  selector:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    component: webhook
---
# Source: vault/templates/injector-deployment.yaml
# Deployment for the injector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-agent-injector
  namespace: vault-example
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
    component: webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault-agent-injector
      app.kubernetes.io/instance: vault
      component: webhook

  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault-agent-injector
        app.kubernetes.io/instance: vault
        component: webhook
    spec:

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/name: vault-agent-injector
                  app.kubernetes.io/instance: "vault"
                  component: webhook
              topologyKey: kubernetes.io/hostname

      serviceAccountName: "vault-agent-injector"

      securityContext:
        runAsNonRoot: true
        runAsGroup: 1000
        runAsUser: 100
        fsGroup: 1000
      hostNetwork: false
      containers:
        - name: sidecar-injector

          image: "hashicorp/vault-k8s:1.1.0"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
          env:
            - name: AGENT_INJECT_LISTEN
              value: :8080
            - name: AGENT_INJECT_LOG_LEVEL
              value: info
            - name: AGENT_INJECT_VAULT_ADDR
              value: "http://10.XXX.YY.ZZZ:31763"
            - name: AGENT_INJECT_VAULT_AUTH_PATH
              value: auth/kubernetes
            - name: AGENT_INJECT_VAULT_IMAGE
              value: "hashicorp/vault:1.12.1"
            - name: AGENT_INJECT_TLS_AUTO
              value: vault-agent-injector-cfg
            - name: AGENT_INJECT_TLS_AUTO_HOSTS
              value: vault-agent-injector-svc,vault-agent-injector-svc.vault-example,vault-agent-injector-svc.vault-example.svc
            - name: AGENT_INJECT_LOG_FORMAT
              value: standard
            - name: AGENT_INJECT_REVOKE_ON_SHUTDOWN
              value: "false"
            - name: AGENT_INJECT_CPU_REQUEST
              value: "250m"
            - name: AGENT_INJECT_CPU_LIMIT
              value: "500m"
            - name: AGENT_INJECT_MEM_REQUEST
              value: "64Mi"
            - name: AGENT_INJECT_MEM_LIMIT
              value: "128Mi"
            - name: AGENT_INJECT_DEFAULT_TEMPLATE
              value: "map"
            - name: AGENT_INJECT_TEMPLATE_CONFIG_EXIT_ON_RETRY_FAILURE
              value: "true"

            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          args:
            - agent-inject
            - 2>&1
          livenessProbe:
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5
---
# Source: vault/templates/injector-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: vault-agent-injector-cfg
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
webhooks:
  - name: vault.hashicorp.com
    failurePolicy: Ignore
    matchPolicy: Exact
    sideEffects: None
    timeoutSeconds: 30
    admissionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:
        name: vault-agent-injector-svc
        namespace: vault-example
        path: "/mutate"
      caBundle: ""
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    objectSelector:
      matchExpressions:
      - key: app.kubernetes.io/name
        operator: NotIn
        values:
        - vault-agent-injector

NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

Upon adding one more clusterrole binding like below fixed the issue. Not sure if this is expected from this helm chart to do the clusterrolebidning out side of helm chart. If thats not the case, it would be great to include this as part of helm chart itself

(⎈ |bge-stg1:vault-example)@~ % cat cluster-role-vault.yaml 
# Source: vault/templates/server-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-server-binding
  labels:
    helm.sh/chart: vault-0.23.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-agent-injector
  namespace: vault-example

Chart version: 0.23.0 K8s Version 1.21

hashicorp / vault-helm

Helm chart with externalVaultAddr doesn't create "vault" service account which has "system:auth-delegator" permission #906