First, thanks for your efforts in creating the plugin. Using Hashicorp Vault as a KMS Provider Plugin for Kubernetes was something I was really waiting for.
Your docs (Getting Started, Configuration) suggest running the plugin as a static Pod. Static Pods cannot reference other API objects such as Secrets or Service Accounts (see kubelet issue on this topic), making it impossible to provide the Vault server CA certificate as a Kubernetes Secret:
Jul 12 11:48:38 n0251 kubelet[1536]: E0712 11:48:38.064303 1536 kubelet.go:1921] "Failed creating a mirror pod for" err="pods \"vault-kubernetes-kms-n0251\" is forbidden: a mirror pod may not reference secrets" pod="kube-system/vault-kubernetes-kms-n0251"
Jul 12 11:48:43 n0251 kubelet[1536]: W0712 11:48:43.648953 1536 reflector.go:539] object-"kube-system"/"vault-kms-ca-cert": failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:48:43 n0251 kubelet[1536]: E0712 11:48:43.648993 1536 reflector.go:147] object-"kube-system"/"vault-kms-ca-cert": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:49:27 n0251 kubelet[1536]: W0712 11:49:27.758976 1536 reflector.go:539] object-"kube-system"/"vault-kms-ca-cert": failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
Jul 12 11:49:27 n0251 kubelet[1536]: E0712 11:49:27.759025 1536 reflector.go:147] object-"kube-system"/"vault-kms-ca-cert": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "vault-kms-ca-cert" is forbidden: User "system:node:n0251" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'n0251' and this object
The behavior makes sense because kubelet spawns static Pods independently of the Kubernetes control-plane availability.
A better solution is to mount the certificate via hostPath. E.g.,
First, thanks for your efforts in creating the plugin. Using Hashicorp Vault as a KMS Provider Plugin for Kubernetes was something I was really waiting for.
Your docs (Getting Started, Configuration) suggest running the plugin as a static Pod. Static Pods cannot reference other API objects such as Secrets or Service Accounts (see kubelet issue on this topic), making it impossible to provide the Vault server CA certificate as a Kubernetes Secret:
The behavior makes sense because
kubelet
spawns static Pods independently of the Kubernetes control-plane availability.A better solution is to mount the certificate via
hostPath
. E.g.,Maybe it's worth reflecting on this in the docs.