Documentation: Caveats when running as static Pod

First, thanks for your efforts in creating the plugin. Using Hashicorp Vault as a KMS Provider Plugin for Kubernetes was something I was really waiting for.

Your docs (Getting Started, Configuration) suggest running the plugin as a static Pod. Static Pods cannot reference other API objects such as Secrets or Service Accounts (see kubelet issue on this topic), making it impossible to provide the Vault server CA certificate as a Kubernetes Secret:

Jul 12 11:48:38 n0251 kubelet[1536]: E0712 11:48:38.064303    1536 kubelet.go:1921] "Failed creating a mirror pod for" err="pods \"vault-kubernetes-kms-n0251\" is forbidden: a mirror pod may not reference secrets" pod="kube-system/vault-kubernetes-kms-n0251"
Jul 12 11:48:43 n0251 kubelet[1536]: W0712 11:48:43.648953    1536 reflector.go:539] object-"kube-system"/"vault-kms-ca-cert": failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:48:43 n0251 kubelet[1536]: E0712 11:48:43.648993    1536 reflector.go:147] object-"kube-system"/"vault-kms-ca-cert": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:49:27 n0251 kubelet[1536]: W0712 11:49:27.758976    1536 reflector.go:539] object-"kube-system"/"vault-kms-ca-cert": failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:49:27 n0251 kubelet[1536]: E0712 11:49:27.759025    1536 reflector.go:147] object-"kube-system"/"vault-kms-ca-cert": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object

The behavior makes sense because kubelet spawns static Pods independently of the Kubernetes control-plane availability.

A better solution is to mount the certificate via hostPath. E.g.,

      env:
        - name: VAULT_CACERT
          value: "/etc/kubernetes/kms/vault-ca.crt"
      volumeMounts:
        - name: kms
          mountPath: /opt/kms
        - name: config
          mountPath: /etc/kubernetes/kms
          readOnly: true
  volumes:
    - name: kms
      hostPath:
        path: /opt/kms
    - name: config
      hostPath:
        path: /etc/kubernetes/kms

Maybe it's worth reflecting on this in the docs.

FalcoSuessgott / vault-kubernetes-kms

Documentation: Caveats when running as static Pod #80