Forbidden response when the vault-agent-injector is enabled in HA mode

Describe the bug

Upon enabling multiple replicas for the vault-agent-injector, we observed a persistent stream of Forbidden responses in the audit log

To Reproduce Steps to reproduce the behavior:

Enable Audit log in Control Plane Logging on AWS EKS dashboard: Configuration > Logging > Audit
Deploy the Helm chart by enabling the injector in HA mode (including multiple replicas and specific affinity rules)
Audit logs

Field   Value
@ingestionTime  1700679623410
@log    <truncated>:/aws/eks/kb-dev/cluster
@logStream  kube-apiserver-audit-e5c60c65e25e60e82d62d99a1c6af6e5
@message    <truncated>
@timestamp  1700679620406
annotations.authorization.k8s.io/decision   forbid
apiVersion  audit.k8s.io/v1
auditID d660d0f9-bce7-4f02-9252-6f3cdba447c4
kind    Event
level   Request
objectRef.apiVersion    v1
objectRef.name  <truncated>
objectRef.resource  nodes
requestReceivedTimestamp    2023-11-22T19:00:20.199897Z
requestURI  /api/v1/nodes/<truncated>
responseStatus.code 403
responseStatus.details.kind nodes
responseStatus.details.name <truncated>
responseStatus.message  nodes "<truncated>" is forbidden: User "system:serviceaccount:kmaas-vault:vault-sidecar-injector-agent-injector" cannot get resource "nodes" in API group "" at the cluster scope
responseStatus.reason   Forbidden
responseStatus.status   Failure
sourceIPs.0 <>
stage   ResponseComplete
stageTimestamp  2023-11-22T19:00:20.201274Z
user.extra.authentication.kubernetes.io/pod-name.0  vault-sidecar-injector-agent-injector-6cb89c7894-mldb8
user.extra.authentication.kubernetes.io/pod-uid.0   0077b46d-e4cb-4d7e-98aa-9c321480af45
user.groups.0   system:serviceaccounts
user.groups.1   system:serviceaccounts:kmaas-vault
user.groups.2   system:authenticated
user.uid    6872dcaa-4ccb-4897-ba6b-9b2cc7d0ba1e
user.username   system:serviceaccount:kmaas-vault:vault-sidecar-injector-agent-injector
userAgent   vault-k8s/v0.0.0 (linux/amd64) kubernetes/$Format
verb    get

Vault injector logs

➜ k get po
NAME                                                     READY   STATUS    RESTARTS   AGE
vault-sidecar-injector-agent-injector-7568478f5c-h5p7f   1/1     Running   0          154m
vault-sidecar-injector-agent-injector-7568478f5c-mtxxz   1/1     Running   0          154m
➜ k logs -f vault-sidecar-injector-agent-injector-7568478f5c-h5p7f
Using internal leader elector logic for webhook certificate management
Registering telemetry path on "/metrics"
Listening on ":8080"...
2023-11-27T09:17:29.803Z [INFO]  handler: Starting handler..
2023-11-27T09:17:29.900Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2023-11-27T09:17:29.901Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.901Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.901Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.901Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.902Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.902Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:29.902Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:34.804Z [INFO]  handler.auto-tls: Currently the leader
2023-11-27T09:17:34.805Z [INFO]  handler.auto-tls: Generated CA
2023-11-27T09:17:34.858Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2023-11-27T09:17:34.871Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:39.799Z [INFO]  handler.auto-tls: Currently the leader
^C
➜ k logs -f vault-sidecar-injector-agent-injector-7568478f5c-mtxxz
Using internal leader elector logic for webhook certificate management
Registering telemetry path on "/metrics"
2023-11-27T09:17:38.310Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2023-11-27T09:17:38.595Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.596Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...
2023-11-27T09:17:38.604Z [INFO]  handler.certwatcher: Webhooks changed. Updating certs...

Expected behavior There should not be any forbidden response in Kubernetes Audit Events from Vault Injector

Environment

Kubernetes version:
- EKS
- v1.27.7-eks-4f4795d
vault-helm version:
- 0.26.1

Chart values:

injector:
  enabled: true
  replicas: 2
  leaderElector:
    enabled: true
  affinity: |
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: {{ template "vault.name" . }}-agent-injector
              app.kubernetes.io/instance: "{{ .Release.Name }}"
              component: webhook
          topologyKey: kubernetes.io/zone

Additional context We managed to resolve it by extending the cluster role to grant 'get' access on 'nodes'

- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get

We've noticed a related closed issue, but it seems the fix or suggestion isn't evident. Therefore, we're raising a new issue to address the matter

hashicorp / vault-helm

Forbidden response when the vault-agent-injector is enabled in HA mode #980