search

ondat / trousseau

Store and access your secrets the Kubernetes native way with any external KMS.
https://trousseau.io
Apache License 2.0
178 stars 11 forks source link

[DEFECT] Missing POD in multi-master context for first master node #245

Open ghost opened 1 year ago

ghost commented 1 year ago

Issue template is used for reporting defects or support issues.

There is no POD on the "FIRST" master node due to lack of toleration specification

Detailed Description

i am experiencing this issue that seems to occur specificly in a multi-master environment.

I deployed a multi-master cluster with 3 master nodes and 4 worker nodes. 2 VMs using haproxy+keepalived share a VIP address to redirect all kubernetes API calls traffic to one of the 3 master nodes behind the haproxy nodes.

During the deployment of the kubernetes cluster, it's important to note it was done on 3 phases:

During installation of trousseau, when applying daemonset manifest i noticed i had only 2 PODS deploying instead of 3.

After some investigations, it seems the FIRST master (the one on which "kubeadm init" was typed) have an additional taint "" that the 2 other control-plane nodes (those on which "kubeadm join --control-plane" was typed) don't have. It seaems that with this additional taint, the "FIRST" master doesn't match the daemonset tolerations criterions and no POD is deployed on this node.

There are 3 problems ordered by gravity with this:

Expected Behavior

A POD is depoyed on every master/control-plane nodes.

Current Behavior

A POD is not depoyed on the "FIRST" master/control-plane node.

Steps to Reproduce

1) Deploy a first master node using "kubeadm init" command 2) Deploy additional master nodes using "kubeadm join" command with "--control-plane" flag 3) Deploy worker nodes "kubeadm join" 4) Deploy Vault and Trousseeau using the methode described in https://github.com/ondat/trousseau/wiki/Trousseau-Deployment#

Context (Environment)

Possible Solution/Implementation

Add an additional toleration:

sysadmin@nisdevkubeadmin01:~/trousseau$ diff trousseau-daemonset.yaml trousseau-daemonset.yaml.ORIGINAL
112,114d111
<         - key: node-role.kubernetes.io/master
<           operator: Exists
<           effect: NoSchedule

Apply it:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl apply -f trousseau-daemonset.yaml
daemonset.apps/vault-kms-provider created

The daemon set now has a value of 3:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe daemonset.apps/vault-kms-provider -n kube-system
Name:           vault-kms-provider
Selector:       name=vault-kms-provider
Node-Selector:  <none>
Labels:         app=vault-kms-provider
                tier=control-plane
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 3
Current Number of Nodes Scheduled: 3
Number of Nodes Scheduled with Up-to-date Pods: 3
Number of Nodes Scheduled with Available Pods: 3
Number of Nodes Misscheduled: 0
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
[...etc...]
Events:
  Type    Reason            Age        From                  Message
  ----    ------            ----       ----                  -------
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-qnkbc
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-klsmf
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-rwl8s

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get daemonset/vault-kms-provider -n kube-system
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vault-kms-provider   3         3         3       3            3           <none>          19s

Possible PR

ghost commented 1 year ago

Logs and elements regarding issue

The cluster nodes:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get nodes -o wide
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
nisdevkubemaster01   Ready    control-plane   47d   v1.25.0   10.2.10.11    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubemaster02   Ready    control-plane   47d   v1.25.0   10.2.10.12    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubemaster03   Ready    control-plane   47d   v1.25.0   10.2.10.13    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker01   Ready    <none>          47d   v1.25.0   10.2.10.21    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker02   Ready    <none>          47d   v1.25.0   10.2.10.22    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker03   Ready    <none>          47d   v1.25.0   10.2.10.23    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2
nisdevkubeworker04   Ready    <none>          47d   v1.25.0   10.2.10.24    <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   docker://23.0.2

The etcd and kube-api pods

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l component=etcd -n kube-system
NAME                      READY   STATUS    RESTARTS      AGE
etcd-nisdevkubemaster01   1/1     Running   2 (51m ago)   47d
etcd-nisdevkubemaster02   1/1     Running   5 (11d ago)   47d
etcd-nisdevkubemaster03   1/1     Running   5 (11d ago)   47d

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l component=kube-apiserver -n kube-system
NAME                                READY   STATUS    RESTARTS   AGE
kube-apiserver-nisdevkubemaster01   1/1     Running   0          15s
kube-apiserver-nisdevkubemaster02   1/1     Running   0          116s
kube-apiserver-nisdevkubemaster03   1/1     Running   0          108s

The specs of the daemonset:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe daemonset.apps/vault-kms-provider -n kube-system
Name:           vault-kms-provider
Selector:       name=vault-kms-provider
Node-Selector:  <none>
Labels:         app=vault-kms-provider
                tier=control-plane
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           name=vault-kms-provider
  Service Account:  vault-auth
  Init Containers:
   vault-agent:
    Image:      vault
    Port:       <none>
    Host Port:  <none>
    Args:
      agent
      -config=/etc/vault/vault-agent-config.hcl
      -log-level=debug
    Environment:
      VAULT_ADDR:  http://10.109.173.136:8200
    Mounts:
      /etc/secrets from shared-data (rw)
      /etc/vault from config (rw)
  Containers:
   vault-kms-provider:
    Image:      ghcr.io/ondat/trousseau:v1.1.3
    Port:       8787/TCP
    Host Port:  8787/TCP
    Args:
      -v=5
      --config-file-path=/opt/trousseau/config.yaml
      --listen-addr=unix:///opt/vault-kms/vaultkms.socket
      --zap-encoder=json
      --v=3
    Limits:
      cpu:     300m
      memory:  256Mi
    Requests:
      cpu:     50m
      memory:  64Mi
    Liveness:  http-get http://:8787/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      VAULT_SKIP_VERIFY:  true
    Mounts:
      /opt/trousseau/ from shared-data (rw)
      /opt/vault-kms from vault-kms (rw)
  Volumes:
   vault-kms:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/vault-kms
    HostPathType:  DirectoryOrCreate
   config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trousseau-vault-agent-config
    Optional:  false
   shared-data:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Events:
  Type    Reason            Age        From                  Message
  ----    ------            ----       ----                  -------
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-l7zpj
  Normal  SuccessfulCreate  <invalid>  daemonset-controller  Created pod: vault-kms-provider-m27bd

The POD deployed:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get pods -l name=vault-kms-provider  -o wide -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
vault-kms-provider-l7zpj   1/1     Running   0          58s   10.2.10.13   nisdevkubemaster03   <none>           <none>
vault-kms-provider-m27bd   1/1     Running   0          58s   10.2.10.12   nisdevkubemaster02   <none>           <none>

The Daemonset states up of 2 desired POD instead of 3:

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get daemonset/vault-kms-provider -n kube-system
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vault-kms-provider   2         2         2       2            2           <none>          2m10s

If encrypting a new secret

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl create secret generic secret-post-deploy -n default --from-literal=mykey=mydata
secret/secret-post-deploy created

It's retrieval can fail depending on which nodes handle the API request regarding the secret:


sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
NAME                 TYPE     DATA   AGE
secret-post-deploy   Opaque   1      2m51s

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
Error from server: illegal base64 data at input byte 3

The error occurs on the node with encryption not activated.

The "FIRST" master throws same kind of errors regarding "illegal base64 data"

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs 87d0b2655ed7
I0515 17:35:34.072788       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:35:34.074047       1 server.go:161] Version: v1.25.0
I0515 17:35:34.074108       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0515 17:35:34.762893       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:35:34.764786       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:34.764810       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:35:34.766571       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:34.766590       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:35:34.835864       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:35:34.837134       1 instance.go:261] Using reconciler: lease
W0515 17:35:34.945556       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:34.945617       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:35.154092       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:35:35.689750       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.691989       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.697785       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:35:35.703823       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:35:35.706087       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.709230       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.709355       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.726067       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.726116       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.732634       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.732693       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.732877       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:35:35.745284       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.745356       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.748957       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.749067       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.758514       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.767660       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:35:35.775217       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:35:35.775259       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:35:35.777914       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.780079       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:35:35.781520       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:35:35.781540       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:35:35.808688       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:35:35.952375       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:35.952451       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:36.959224       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:36.959343       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:37.458302       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:35:37.458303       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:35:37.458766       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:35:37.458849       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:35:37.459986       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:35:37.460042       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:35:37.460118       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:35:37.466501       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:35:37.466535       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:35:37.466572       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:35:37.466577       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:35:37.467258       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:35:37.475120       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:35:37.475328       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:35:37.476453       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:35:37.476525       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:35:37.476595       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:35:37.476852       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:35:37.476892       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
E0515 17:35:37.513790       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:37.515436       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:37.515504       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
I0515 17:35:37.476977       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:35:37.479710       1 controller.go:85] Starting OpenAPI controller
I0515 17:35:37.479744       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:35:37.479771       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:35:37.479799       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:35:37.479818       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:35:37.479833       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:35:37.479850       1 crd_finalizer.go:266] Starting CRDFinalizer
I0515 17:35:37.486404       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:35:37.524909       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
I0515 17:35:37.560203       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:35:37.563100       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:35:37.566703       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:35:37.567239       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:35:37.578330       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:35:37.578482       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:35:37.628420       1 shared_informer.go:262] Caches are synced for crd-autoregister
W0515 17:35:37.964107       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:37.964178       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
I0515 17:35:38.114255       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0515 17:35:38.487235       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
E0515 17:35:38.579921       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:38.580766       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:38.580906       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:38.967167       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:38.967221       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:39.970543       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:39.970573       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
E0515 17:35:40.739566       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:40.740581       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:40.740638       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:40.975008       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:40.975578       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:41.979366       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:41.979466       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:42.983733       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:42.983839       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:43.987062       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:43.987161       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
W0515 17:35:44.990256       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
E0515 17:35:44.990284       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: illegal base64 data at input byte 3; reinitializing...
E0515 17:35:45.729395       1 status.go:71] apiserver received an error that is not an metav1.Status: 3: illegal base64 data at input byte 3
W0515 17:35:45.730317       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: illegal base64 data at input byte 3
E0515 17:35:45.730483       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: illegal base64 data at input byte 3
W0515 17:35:45.993650       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: illegal base64 data at input byte 3
[...etc...]

When activating the encryption on the "FIRST" master using the following manifest, new errors are thrown:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.2.10.11:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
[...]
    - --encryption-provider-config=/etc/kubernetes/custom/encryption-config.yaml
    - mountPath: /etc/kubernetes/custom
      name: custom-config
      readOnly: true
    - mountPath: /opt/vault-kms/
      name: kms-socket
      readOnly: false
[...]
  volumes:
[...]
  - hostPath:
      path: /etc/kubernetes/custom
      type: DirectoryOrCreate
    name: custom-config
[...]
  - hostPath:
      path: /opt/vault-kms
      type: DirectoryOrCreate
    name: kms-socket

The new errors thrown refers to issue with communication with gRPC server using socket file :

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs cc3db8189c01
I0515 17:42:16.157636       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:42:16.158614       1 server.go:161] Version: v1.25.0
I0515 17:42:16.158667       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E0515 17:42:16.646681       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:16.646889       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:16.647202       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:16.647293       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:16.648287       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:42:16.650018       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:16.650041       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:42:16.651554       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:16.651571       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:42:16.725046       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:42:16.727641       1 instance.go:261] Using reconciler: lease
I0515 17:42:17.029848       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:42:17.551934       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.554766       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.560536       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:42:17.566833       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:42:17.569340       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.573850       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.574112       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.584711       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.584743       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.592227       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.592266       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.592552       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:42:17.608699       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.608740       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.611583       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.611617       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.618695       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.625013       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:42:17.631577       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:42:17.631614       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:42:17.634346       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:42:17.636618       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:42:17.638131       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:42:17.638157       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
E0515 17:42:17.647716       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:17.647870       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
W0515 17:42:17.665928       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
I0515 17:42:19.357654       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:42:19.357873       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:42:19.376162       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:42:19.376807       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:42:19.376902       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:42:19.378304       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:42:19.378355       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:42:19.378884       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:42:19.381143       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:42:19.390073       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:42:19.390334       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:42:19.390808       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:42:19.390827       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:42:19.390862       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:42:19.390906       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:42:19.390918       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:42:19.390956       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:42:19.390966       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:42:19.394609       1 controller.go:85] Starting OpenAPI controller
I0515 17:42:19.394675       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:42:19.394701       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:42:19.394751       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:42:19.394774       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:42:19.394800       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:42:19.394887       1 crd_finalizer.go:266] Starting CRDFinalizer
I0515 17:42:19.395282       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:42:19.399703       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:42:19.401355       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:42:19.401373       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
E0515 17:42:19.431444       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:19.431662       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:19.448569       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:42:19.479595       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:42:19.490948       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:42:19.491143       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:42:19.491742       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:42:19.491781       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:42:19.506631       1 shared_informer.go:262] Caches are synced for crd-autoregister
E0515 17:42:19.549355       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:19.549582       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:19.808223       1 trace.go:205] Trace[1443041845]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:16.797) (total time: 3010ms):
Trace[1443041845]: [3.010156003s] [3.010156003s] END
W0515 17:42:19.808574       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:19.808662       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:20.392012       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
I0515 17:42:20.547441       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
E0515 17:42:21.202436       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:21.202563       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:21.856333       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:21.856433       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:22.421060       1 trace.go:205] Trace[995690186]: "List(recursive=true) etcd3" audit-id:1fc8fb24-bb4b-4409-a56f-fa7b2e9b10db,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:19.395) (total time: 3025ms):
Trace[995690186]: [3.025995675s] [3.025995675s] END
I0515 17:42:22.421257       1 trace.go:205] Trace[730370234]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:1fc8fb24-bb4b-4409-a56f-fa7b2e9b10db,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:19.394) (total time: 3026ms):
Trace[730370234]: [3.026263657s] [3.026263657s] END
W0515 17:42:22.422844       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:22.423190       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:23.815578       1 trace.go:205] Trace[1183215355]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:20.809) (total time: 3005ms):
Trace[1183215355]: [3.005657272s] [3.005657272s] END
W0515 17:42:23.815635       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:23.815679       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:25.436566       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:25.436634       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:25.526406       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:25.526592       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:26.863508       1 trace.go:205] Trace[541467408]: "List(recursive=true) etcd3" audit-id:31c9c553-aa87-495f-a05e-a7524e7a4948,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:23.858) (total time: 3005ms):
Trace[541467408]: [3.005291123s] [3.005291123s] END
I0515 17:42:26.863998       1 trace.go:205] Trace[914437271]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:31c9c553-aa87-495f-a05e-a7524e7a4948,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:23.858) (total time: 3005ms):
Trace[914437271]: [3.005834219s] [3.005834219s] END
W0515 17:42:26.864788       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:26.865131       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:27.818852       1 trace.go:205] Trace[2013110149]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:24.816) (total time: 3002ms):
Trace[2013110149]: [3.002695001s] [3.002695001s] END
W0515 17:42:27.818903       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:27.818920       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:30.910587       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:30.910938       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:31.437922       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:31.437987       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:31.823964       1 trace.go:205] Trace[1637434811]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:28.819) (total time: 3004ms):
Trace[1637434811]: [3.004718217s] [3.004718217s] END
W0515 17:42:31.824006       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:31.824100       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:32.310942       1 trace.go:205] Trace[1765368075]: "List(recursive=true) etcd3" audit-id:892c4e2c-8519-468a-9d33-c71f84e2ad79,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:29.308) (total time: 3002ms):
Trace[1765368075]: [3.002766092s] [3.002766092s] END
I0515 17:42:32.311074       1 trace.go:205] Trace[860303514]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:892c4e2c-8519-468a-9d33-c71f84e2ad79,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:29.308) (total time: 3002ms):
Trace[860303514]: [3.002944476s] [3.002944476s] END
W0515 17:42:32.311594       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:32.311623       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:35.827724       1 trace.go:205] Trace[5852664]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:32.824) (total time: 3002ms):
Trace[5852664]: [3.002953826s] [3.002953826s] END
W0515 17:42:35.827852       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:35.827889       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:42:38.011513       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:42:38.011628       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:42:39.833073       1 trace.go:205] Trace[313620331]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:42:36.828) (total time: 3004ms):
Trace[313620331]: [3.004095391s] [3.004095391s] END
W0515 17:42:39.833220       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:39.833263       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:42:39.946339       1 trace.go:205] Trace[160050274]: "List(recursive=true) etcd3" audit-id:ee5c6eae-099a-4d90-95ea-4716bcaead40,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:42:36.943) (total time: 3002ms):
Trace[160050274]: [3.002653112s] [3.002653112s] END
I0515 17:42:39.946610       1 trace.go:205] Trace[2133478831]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ee5c6eae-099a-4d90-95ea-4716bcaead40,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:42:36.943) (total time: 3002ms):
Trace[2133478831]: [3.002964763s] [3.002964763s] END
W0515 17:42:39.947143       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:42:39.947177       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
[...etc...]

Furthermore retrieval of secrets still generated issue (regarding the socket communication):

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl get secret -n default secret-post-deploy
Error from server (InternalError): Internal error occurred: failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"

If we look more at node description, we can notice the "FIRST" master avec 2 Taints: "node-role.kubernetes.io/control-plane:NoSchedule" and "node-role.kubernetes.io/master:NoSchedule".

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster01
Name:               nisdevkubemaster01
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 18:59:03 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
                    node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster02
Name:               nisdevkubemaster02
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster02
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 19:00:31 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false

sysadmin@nisdevkubeadmin01:~/trousseau$ kubectl describe node nisdevkubemaster03
Name:               nisdevkubemaster03
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=nisdevkubemaster03
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 28 Mar 2023 19:00:30 +0200
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false

After a certain time, the "FIRST" master kube-apiserver POD is exited and restarted

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker ps -a | grep -i kube-api
80e92655e694   4d2edfd10d3e                "kube-apiserver --ad…"   2 minutes ago       Up 2 minutes                             k8s_kube-apiserver_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_2
0ff23e9cd858   4d2edfd10d3e                "kube-apiserver --ad…"   6 minutes ago       Exited (137) 2 minutes ago               k8s_kube-apiserver_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_1
74d59a631d86   registry.k8s.io/pause:3.6   "/pause"                 11 minutes ago      Up 11 minutes                            k8s_POD_kube-apiserver-nisdevkubemaster01_kube-system_bfe2ce82dc21b1574a9e6dc2e7e65fbe_0

root@nisdevkubemaster01:/etc/kubernetes/manifests# docker logs 0ff23e9cd858
I0515 17:46:56.058530       1 server.go:563] external host was not specified, using 10.2.10.11
I0515 17:46:56.059182       1 server.go:161] Version: v1.25.0
I0515 17:46:56.059246       1 server.go:163] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
E0515 17:46:56.782143       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
E0515 17:46:56.782147       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:56.782243       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
W0515 17:46:56.782261       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:56.782341       1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
I0515 17:46:56.782768       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:56.782780       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0515 17:46:56.784095       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:56.784119       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:46:56.840202       1 genericapiserver.go:656] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
I0515 17:46:56.841432       1 instance.go:261] Using reconciler: lease
I0515 17:46:57.089775       1 instance.go:574] API group "internal.apiserver.k8s.io" is not enabled, skipping.
W0515 17:46:57.536347       1 genericapiserver.go:656] Skipping API authentication.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.544060       1 genericapiserver.go:656] Skipping API authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.568767       1 genericapiserver.go:656] Skipping API autoscaling/v2beta1 because it has no resources.
W0515 17:46:57.574544       1 genericapiserver.go:656] Skipping API batch/v1beta1 because it has no resources.
W0515 17:46:57.576934       1 genericapiserver.go:656] Skipping API certificates.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.579022       1 genericapiserver.go:656] Skipping API coordination.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.579086       1 genericapiserver.go:656] Skipping API discovery.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.583644       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.583664       1 genericapiserver.go:656] Skipping API networking.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.585327       1 genericapiserver.go:656] Skipping API node.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.585343       1 genericapiserver.go:656] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.585390       1 genericapiserver.go:656] Skipping API policy/v1beta1 because it has no resources.
W0515 17:46:57.589827       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.589850       1 genericapiserver.go:656] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.591475       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.591495       1 genericapiserver.go:656] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.596328       1 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.600822       1 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
W0515 17:46:57.606039       1 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources.
W0515 17:46:57.606061       1 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources.
W0515 17:46:57.607975       1 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources.
W0515 17:46:57.609603       1 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources.
I0515 17:46:57.610697       1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0515 17:46:57.610712       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
W0515 17:46:57.637286       1 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources.
E0515 17:46:57.783174       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:57.783245       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:46:59.104994       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:59.105093       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:59.159717       1 secure_serving.go:210] Serving securely on [::]:6443
I0515 17:46:59.159870       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:46:59.161782       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:46:59.164069       1 autoregister_controller.go:141] Starting autoregister controller
I0515 17:46:59.164202       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0515 17:46:59.165241       1 apf_controller.go:300] Starting API Priority and Fairness config controller
I0515 17:46:59.165397       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:46:59.165570       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0515 17:46:59.167261       1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0515 17:46:59.167290       1 shared_informer.go:255] Waiting for caches to sync for cluster_authentication_trust_controller
I0515 17:46:59.169569       1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0515 17:46:59.169586       1 shared_informer.go:255] Waiting for caches to sync for crd-autoregister
I0515 17:46:59.169809       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:46:59.189651       1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:46:59.190583       1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:46:59.193829       1 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0515 17:46:59.196657       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0515 17:46:59.196470       1 available_controller.go:491] Starting AvailableConditionController
I0515 17:46:59.197268       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0515 17:46:59.197337       1 controller.go:80] Starting OpenAPI V3 AggregationController
I0515 17:46:59.197659       1 controller.go:83] Starting OpenAPI AggregationController
I0515 17:46:59.198127       1 customresource_discovery_controller.go:209] Starting DiscoveryController
I0515 17:46:59.199915       1 controller.go:85] Starting OpenAPI controller
I0515 17:46:59.200020       1 controller.go:85] Starting OpenAPI V3 controller
I0515 17:46:59.200091       1 naming_controller.go:291] Starting NamingConditionController
I0515 17:46:59.200160       1 establishing_controller.go:76] Starting EstablishingController
I0515 17:46:59.200230       1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
I0515 17:46:59.200265       1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:46:59.200320       1 crd_finalizer.go:266] Starting CRDFinalizer
E0515 17:46:59.288402       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:46:59.288527       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:46:59.297543       1 cache.go:39] Caches are synced for AvailableConditionController controller
I0515 17:46:59.297557       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0515 17:46:59.380140       1 cache.go:39] Caches are synced for autoregister controller
I0515 17:46:59.381196       1 shared_informer.go:262] Caches are synced for cluster_authentication_trust_controller
I0515 17:46:59.381355       1 shared_informer.go:262] Caches are synced for crd-autoregister
I0515 17:46:59.383050       1 shared_informer.go:262] Caches are synced for node_authorizer
I0515 17:46:59.386369       1 apf_controller.go:305] Running API Priority and Fairness config worker
I0515 17:46:59.879843       1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0515 17:46:59.902711       1 trace.go:205] Trace[41463727]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:46:56.897) (total time: 3005ms):
Trace[41463727]: [3.005633161s] [3.005633161s] END
W0515 17:46:59.903173       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:46:59.903355       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:00.167691       1 storage_scheduling.go:111] all system priority classes are created successfully or already exist.
E0515 17:47:01.025758       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:01.025903       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:01.944001       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:01.944083       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:02.279646       1 trace.go:205] Trace[284735070]: "List(recursive=true) etcd3" audit-id:ba3219cb-566b-48b2-93ee-480bdef2ac06,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:46:59.270) (total time: 3008ms):
Trace[284735070]: [3.008960631s] [3.008960631s] END
I0515 17:47:02.280014       1 trace.go:205] Trace[2055471106]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ba3219cb-566b-48b2-93ee-480bdef2ac06,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:46:59.270) (total time: 3009ms):
Trace[2055471106]: [3.009334798s] [3.009334798s] END
W0515 17:47:02.280715       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:02.280840       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:03.907210       1 trace.go:205] Trace[2042703226]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:00.903) (total time: 3003ms):
Trace[2042703226]: [3.003285857s] [3.003285857s] END
W0515 17:47:03.907316       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:03.907546       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:05.296515       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:05.296698       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:06.161563       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:06.161636       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:06.526685       1 trace.go:205] Trace[730767454]: "List(recursive=true) etcd3" audit-id:ce524b37-6fed-4477-a9bf-dca49887e782,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:03.523) (total time: 3003ms):
Trace[730767454]: [3.003023214s] [3.003023214s] END
I0515 17:47:06.527254       1 trace.go:205] Trace[2058701975]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:ce524b37-6fed-4477-a9bf-dca49887e782,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:03.523) (total time: 3003ms):
Trace[2058701975]: [3.003562365s] [3.003562365s] END
W0515 17:47:06.528416       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:06.528457       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:07.911678       1 trace.go:205] Trace[1192110976]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:04.908) (total time: 3003ms):
Trace[1192110976]: [3.003093004s] [3.003093004s] END
W0515 17:47:07.911821       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:07.911902       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:11.394880       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:11.395074       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:11.757641       1 trace.go:205] Trace[619589245]: "List(recursive=true) etcd3" audit-id:9fc0e54c-2bce-462f-8687-dd5bb4e8d2d4,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:08.754) (total time: 3003ms):
Trace[619589245]: [3.003093465s] [3.003093465s] END
I0515 17:47:11.757763       1 trace.go:205] Trace[745073040]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:9fc0e54c-2bce-462f-8687-dd5bb4e8d2d4,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:08.754) (total time: 3003ms):
Trace[745073040]: [3.003258404s] [3.003258404s] END
W0515 17:47:11.758294       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:11.758343       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:11.915617       1 trace.go:205] Trace[350332463]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:08.912) (total time: 3003ms):
Trace[350332463]: [3.003056785s] [3.003056785s] END
W0515 17:47:11.915661       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:11.915678       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:13.880230       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:13.880601       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:15.924355       1 trace.go:205] Trace[1803151140]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:12.916) (total time: 3007ms):
Trace[1803151140]: [3.007638796s] [3.007638796s] END
W0515 17:47:15.924471       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:15.924503       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:18.917633       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:18.917723       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:19.930171       1 trace.go:205] Trace[961899593]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:16.925) (total time: 3004ms):
Trace[961899593]: [3.00455736s] [3.00455736s] END
W0515 17:47:19.930262       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:19.930289       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:20.156403       1 trace.go:205] Trace[1041693117]: "List(recursive=true) etcd3" audit-id:a50c74de-05f0-47b6-9da5-0f62c9c3cbee,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:17.152) (total time: 3003ms):
Trace[1041693117]: [3.003846681s] [3.003846681s] END
I0515 17:47:20.156586       1 trace.go:205] Trace[1219087519]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:a50c74de-05f0-47b6-9da5-0f62c9c3cbee,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:17.152) (total time: 3004ms):
Trace[1219087519]: [3.00409859s] [3.00409859s] END
W0515 17:47:20.157050       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:20.157090       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:23.846723       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:23.846996       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:23.935202       1 trace.go:205] Trace[669387932]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:20.931) (total time: 3003ms):
Trace[669387932]: [3.003739883s] [3.003739883s] END
W0515 17:47:23.935259       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:23.935274       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:27.941235       1 trace.go:205] Trace[1458727689]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:24.935) (total time: 3005ms):
Trace[1458727689]: [3.005547556s] [3.005547556s] END
W0515 17:47:27.941283       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:27.941296       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:29.695725       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:29.695887       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:31.945679       1 trace.go:205] Trace[1144863289]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:28.941) (total time: 3004ms):
Trace[1144863289]: [3.004178676s] [3.004178676s] END
W0515 17:47:31.945732       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:31.945769       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:35.451608       1 trace.go:205] Trace[1834458678]: "List(recursive=true) etcd3" audit-id:6417048f-cd2f-4f68-ba11-2184f2845129,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:32.447) (total time: 3004ms):
Trace[1834458678]: [3.004312645s] [3.004312645s] END
I0515 17:47:35.451797       1 trace.go:205] Trace[73531258]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:6417048f-cd2f-4f68-ba11-2184f2845129,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:32.447) (total time: 3004ms):
Trace[73531258]: [3.004528771s] [3.004528771s] END
W0515 17:47:35.452286       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:35.452343       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:35.951913       1 trace.go:205] Trace[353802577]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:32.946) (total time: 3005ms):
Trace[353802577]: [3.005047047s] [3.005047047s] END
W0515 17:47:35.951972       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:35.951987       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:37.896794       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:37.896942       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:39.956952       1 trace.go:205] Trace[1720376311]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:36.952) (total time: 3003ms):
Trace[1720376311]: [3.00397061s] [3.00397061s] END
W0515 17:47:39.957001       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:39.957015       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:43.962075       1 trace.go:205] Trace[732546678]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:40.958) (total time: 3004ms):
Trace[732546678]: [3.004016272s] [3.004016272s] END
W0515 17:47:43.962138       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:43.962152       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:47:47.896908       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:47:47.897092       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:47.966712       1 trace.go:205] Trace[466044327]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:44.962) (total time: 3003ms):
Trace[466044327]: [3.003962077s] [3.003962077s] END
W0515 17:47:47.966871       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:47.966911       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:51.971314       1 trace.go:205] Trace[152847791]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:48.967) (total time: 3003ms):
Trace[152847791]: [3.00361396s] [3.00361396s] END
W0515 17:47:51.971366       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:51.971382       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:55.975350       1 trace.go:205] Trace[548900740]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:52.971) (total time: 3003ms):
Trace[548900740]: [3.003577628s] [3.003577628s] END
W0515 17:47:55.975507       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:55.975628       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:47:58.656782       1 trace.go:205] Trace[1204988067]: "List(recursive=true) etcd3" audit-id:7246c517-f979-46df-98f3-91e50385d9fc,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:47:55.652) (total time: 3003ms):
Trace[1204988067]: [3.003753233s] [3.003753233s] END
I0515 17:47:58.656919       1 trace.go:205] Trace[1807520032]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:7246c517-f979-46df-98f3-91e50385d9fc,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:47:55.652) (total time: 3003ms):
Trace[1807520032]: [3.003919313s] [3.003919313s] END
W0515 17:47:58.657616       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:58.657882       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:47:59.980923       1 trace.go:205] Trace[1666681437]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:47:56.976) (total time: 3004ms):
Trace[1666681437]: [3.004235707s] [3.004235707s] END
W0515 17:47:59.981023       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:47:59.981051       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:03.985857       1 trace.go:205] Trace[1780984614]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:00.981) (total time: 3004ms):
Trace[1780984614]: [3.004008418s] [3.004008418s] END
W0515 17:48:03.985952       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:03.985988       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:07.994368       1 trace.go:205] Trace[441807241]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:04.986) (total time: 3007ms):
Trace[441807241]: [3.007650779s] [3.007650779s] END
W0515 17:48:07.994425       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:07.994448       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:09.232762       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:09.232841       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:11.997734       1 trace.go:205] Trace[814431104]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:08.994) (total time: 3002ms):
Trace[814431104]: [3.002826481s] [3.002826481s] END
W0515 17:48:11.997779       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:11.997829       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:16.001887       1 trace.go:205] Trace[1968403247]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:12.997) (total time: 3003ms):
Trace[1968403247]: [3.003822832s] [3.003822832s] END
W0515 17:48:16.001954       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:16.002472       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:18.295272       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:18.295583       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:20.005099       1 trace.go:205] Trace[711726798]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:17.002) (total time: 3002ms):
Trace[711726798]: [3.00240867s] [3.00240867s] END
W0515 17:48:20.005195       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:20.005227       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:24.009309       1 trace.go:205] Trace[490107260]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:21.005) (total time: 3003ms):
Trace[490107260]: [3.003591429s] [3.003591429s] END
W0515 17:48:24.009363       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:24.009380       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:28.015116       1 trace.go:205] Trace[389252337]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:25.010) (total time: 3004ms):
Trace[389252337]: [3.004553925s] [3.004553925s] END
W0515 17:48:28.015160       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:28.015177       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:32.020501       1 trace.go:205] Trace[1750352782]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:29.016) (total time: 3004ms):
Trace[1750352782]: [3.004376719s] [3.004376719s] END
W0515 17:48:32.020554       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:32.020573       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:36.025053       1 trace.go:205] Trace[1873891737]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:33.021) (total time: 3003ms):
Trace[1873891737]: [3.003544018s] [3.003544018s] END
W0515 17:48:36.025110       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:36.025171       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:40.030964       1 trace.go:205] Trace[1978618791]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:37.026) (total time: 3004ms):
Trace[1978618791]: [3.004201314s] [3.004201314s] END
W0515 17:48:40.031018       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:40.031032       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:44.035805       1 trace.go:205] Trace[206298777]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:41.031) (total time: 3004ms):
Trace[206298777]: [3.004347769s] [3.004347769s] END
W0515 17:48:44.035854       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:44.035872       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:44.688191       1 trace.go:205] Trace[931530703]: "List(recursive=true) etcd3" audit-id:4d6105b8-65e3-4bb6-bd51-6c219c3945f2,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:48:41.685) (total time: 3002ms):
Trace[931530703]: [3.002950201s] [3.002950201s] END
I0515 17:48:44.688394       1 trace.go:205] Trace[812410954]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:4d6105b8-65e3-4bb6-bd51-6c219c3945f2,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:48:41.685) (total time: 3003ms):
Trace[812410954]: [3.003192572s] [3.003192572s] END
W0515 17:48:44.689163       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:44.689414       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:48:48.040105       1 trace.go:205] Trace[147757836]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:45.036) (total time: 3003ms):
Trace[147757836]: [3.003306505s] [3.003306505s] END
W0515 17:48:48.040228       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:48.040281       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:52.045261       1 trace.go:205] Trace[2037452132]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:49.040) (total time: 3004ms):
Trace[2037452132]: [3.004299925s] [3.004299925s] END
W0515 17:48:52.045317       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:52.045332       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:48:56.051162       1 trace.go:205] Trace[1123320100]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:53.045) (total time: 3005ms):
Trace[1123320100]: [3.005122395s] [3.005122395s] END
W0515 17:48:56.051249       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:56.051297       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:48:57.769721       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:57.769868       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:48:57.876917       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:48:57.877027       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:00.055653       1 trace.go:205] Trace[1096111447]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:48:57.051) (total time: 3003ms):
Trace[1096111447]: [3.003920014s] [3.003920014s] END
W0515 17:49:00.055788       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:00.055826       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:04.063245       1 trace.go:205] Trace[952464393]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:01.056) (total time: 3007ms):
Trace[952464393]: [3.007105596s] [3.007105596s] END
W0515 17:49:04.063315       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:04.063368       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:08.067562       1 trace.go:205] Trace[1379417391]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:05.064) (total time: 3003ms):
Trace[1379417391]: [3.003355436s] [3.003355436s] END
W0515 17:49:08.067650       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:08.067751       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:12.071789       1 trace.go:205] Trace[836301313]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:09.068) (total time: 3002ms):
Trace[836301313]: [3.002980546s] [3.002980546s] END
W0515 17:49:12.071841       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:12.071855       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:16.076365       1 trace.go:205] Trace[597938773]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:13.072) (total time: 3003ms):
Trace[597938773]: [3.003710619s] [3.003710619s] END
W0515 17:49:16.076414       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:16.076602       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:20.081414       1 trace.go:205] Trace[1701438819]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:17.077) (total time: 3003ms):
Trace[1701438819]: [3.003677276s] [3.003677276s] END
W0515 17:49:20.081487       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:20.081506       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:24.085053       1 trace.go:205] Trace[1036974030]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:21.082) (total time: 3002ms):
Trace[1036974030]: [3.002925632s] [3.002925632s] END
W0515 17:49:24.085092       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:24.085145       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:28.090134       1 trace.go:205] Trace[1187718898]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:25.085) (total time: 3004ms):
Trace[1187718898]: [3.004364001s] [3.004364001s] END
W0515 17:49:28.090422       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:28.090521       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:32.094174       1 trace.go:205] Trace[118669017]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:29.090) (total time: 3003ms):
Trace[118669017]: [3.003385054s] [3.003385054s] END
W0515 17:49:32.094232       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:32.094247       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:35.673699       1 trace.go:205] Trace[1825072249]: "List(recursive=true) etcd3" audit-id:7a1b9680-5ff4-4dc5-865f-604d15ab6dd4,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:49:32.669) (total time: 3003ms):
Trace[1825072249]: [3.003719034s] [3.003719034s] END
I0515 17:49:35.673817       1 trace.go:205] Trace[1409462059]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:7a1b9680-5ff4-4dc5-865f-604d15ab6dd4,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:49:32.669) (total time: 3003ms):
Trace[1409462059]: [3.003886146s] [3.003886146s] END
W0515 17:49:35.674398       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:35.674656       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:36.097032       1 trace.go:205] Trace[1958951863]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:33.094) (total time: 3002ms):
Trace[1958951863]: [3.002615148s] [3.002615148s] END
W0515 17:49:36.097070       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:36.097083       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:40.100709       1 trace.go:205] Trace[294960884]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:37.097) (total time: 3003ms):
Trace[294960884]: [3.003083128s] [3.003083128s] END
W0515 17:49:40.100758       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:40.100775       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:44.103671       1 trace.go:205] Trace[2104243303]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:41.101) (total time: 3002ms):
Trace[2104243303]: [3.002082077s] [3.002082077s] END
W0515 17:49:44.103797       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:44.103827       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:48.108662       1 trace.go:205] Trace[470392946]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:45.104) (total time: 3004ms):
Trace[470392946]: [3.004146033s] [3.004146033s] END
W0515 17:49:48.108737       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:48.108754       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:49:52.113106       1 trace.go:205] Trace[752840887]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:49.109) (total time: 3003ms):
Trace[752840887]: [3.003751281s] [3.003751281s] END
W0515 17:49:52.113157       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:52.113174       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:49:54.541978       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:49:54.542061       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:49:56.117416       1 trace.go:205] Trace[1281744931]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:53.113) (total time: 3003ms):
Trace[1281744931]: [3.003416608s] [3.003416608s] END
W0515 17:49:56.117517       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:49:56.117539       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:00.122283       1 trace.go:205] Trace[529705241]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:49:57.118) (total time: 3003ms):
Trace[529705241]: [3.003648513s] [3.003648513s] END
W0515 17:50:00.122336       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:00.122352       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
E0515 17:50:02.885240       1 grpc_service.go:75] failed to create connection to unix socket: /opt/vault-kms/vaultkms.socket, error: dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory
W0515 17:50:02.885419       1 logging.go:59] [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
  "Addr": "/opt/vault-kms/vaultkms.socket",
  "ServerName": "/opt/vault-kms/vaultkms.socket",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:50:04.127379       1 trace.go:205] Trace[970261590]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:01.123) (total time: 3003ms):
Trace[970261590]: [3.003998927s] [3.003998927s] END
W0515 17:50:04.127480       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:04.127834       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:08.133833       1 trace.go:205] Trace[654433592]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:05.128) (total time: 3004ms):
Trace[654433592]: [3.004976274s] [3.004976274s] END
W0515 17:50:08.134082       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:08.134171       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:12.137953       1 trace.go:205] Trace[464761580]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:09.134) (total time: 3003ms):
Trace[464761580]: [3.003587332s] [3.003587332s] END
W0515 17:50:12.138000       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:12.138019       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:14.450160       1 trace.go:205] Trace[1262963168]: "Get" url:/api/v1/namespaces/default/secrets/secret-post-deploy,user-agent:kubectl/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:fd988e33-8946-4c60-855f-f523d8c0c547,client:10.2.10.3,accept:application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json,protocol:HTTP/2.0 (15-May-2023 17:50:11.447) (total time: 3002ms):
Trace[1262963168]: [3.002631314s] [3.002631314s] END
I0515 17:50:16.143390       1 trace.go:205] Trace[636923892]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:13.139) (total time: 3004ms):
Trace[636923892]: [3.004134024s] [3.004134024s] END
W0515 17:50:16.143474       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:16.143500       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:20.149658       1 trace.go:205] Trace[684083666]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:17.144) (total time: 3005ms):
Trace[684083666]: [3.005003758s] [3.005003758s] END
W0515 17:50:20.149738       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:20.149768       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:24.154513       1 trace.go:205] Trace[1678743422]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:21.150) (total time: 3004ms):
Trace[1678743422]: [3.004155949s] [3.004155949s] END
W0515 17:50:24.154586       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:24.154609       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:24.766317       1 trace.go:205] Trace[195294388]: "Get" url:/api/v1/namespaces/default/secrets/secret-post-deploy,user-agent:kubectl/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:aea0ae66-2ba2-4405-a3f0-4f3b180e70e1,client:10.2.10.3,accept:application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json,protocol:HTTP/2.0 (15-May-2023 17:50:21.762) (total time: 3003ms):
Trace[195294388]: [3.003868712s] [3.003868712s] END
I0515 17:50:28.159887       1 trace.go:205] Trace[1589446782]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:25.155) (total time: 3004ms):
Trace[1589446782]: [3.004832279s] [3.004832279s] END
W0515 17:50:28.159940       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:28.159959       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:28.260761       1 trace.go:205] Trace[1933186184]: "List(recursive=true) etcd3" audit-id:5197abbb-c335-4216-8938-6f0f002499aa,key:/secrets,resourceVersion:0,resourceVersionMatch:,limit:500,continue: (15-May-2023 17:50:25.255) (total time: 3004ms):
Trace[1933186184]: [3.004832083s] [3.004832083s] END
I0515 17:50:28.261200       1 trace.go:205] Trace[2106224912]: "List" url:/api/v1/secrets,user-agent:kube-apiserver/v1.25.0 (linux/amd64) kubernetes/a866cbe,audit-id:5197abbb-c335-4216-8938-6f0f002499aa,client:::1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-May-2023 17:50:25.255) (total time: 3005ms):
Trace[2106224912]: [3.005344858s] [3.005344858s] END
W0515 17:50:28.262106       1 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:28.262155       1 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: failed to list *v1.Secret: Internal error occurred: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
I0515 17:50:32.165829       1 trace.go:205] Trace[344162878]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:29.161) (total time: 3004ms):
Trace[344162878]: [3.00459021s] [3.00459021s] END
W0515 17:50:32.165896       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:32.165968       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:36.170380       1 trace.go:205] Trace[355063030]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:33.166) (total time: 3004ms):
Trace[355063030]: [3.004035253s] [3.004035253s] END
W0515 17:50:36.170451       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:36.170528       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:40.175222       1 trace.go:205] Trace[1268436977]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:37.170) (total time: 3004ms):
Trace[1268436977]: [3.004226319s] [3.004226319s] END
W0515 17:50:40.175287       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:40.175306       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:44.180048       1 trace.go:205] Trace[1508039681]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:41.175) (total time: 3004ms):
Trace[1508039681]: [3.004068572s] [3.004068572s] END
W0515 17:50:44.180220       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:44.180268       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:48.185920       1 trace.go:205] Trace[1825445912]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:45.181) (total time: 3004ms):
Trace[1825445912]: [3.004743747s] [3.004743747s] END
W0515 17:50:48.186052       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:48.186079       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:52.192550       1 trace.go:205] Trace[16415966]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:49.187) (total time: 3004ms):
Trace[16415966]: [3.004834147s] [3.004834147s] END
W0515 17:50:52.192607       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:52.192626       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:50:55.747708       1 controller.go:211] Shutting down kubernetes service endpoint reconciler
E0515 17:50:55.751203       1 controller.go:214] Unable to remove endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.2.10.11, ResourceVersion: 0, AdditionalErrorMsg:
I0515 17:50:55.751604       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.751738       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.751779       1 controller.go:157] Shutting down quota evaluator
I0515 17:50:55.753007       1 controller.go:115] Shutting down OpenAPI V3 controller
I0515 17:50:55.753185       1 controller.go:122] Shutting down OpenAPI controller
I0515 17:50:55.753272       1 apf_controller.go:309] Shutting down API Priority and Fairness config worker
I0515 17:50:55.753362       1 storage_flowcontrol.go:172] APF bootstrap ensurer is exiting
I0515 17:50:55.753449       1 crdregistration_controller.go:142] Shutting down crd-autoregister controller
I0515 17:50:55.753545       1 autoregister_controller.go:165] Shutting down autoregister controller
I0515 17:50:55.753628       1 apiservice_controller.go:131] Shutting down APIServiceRegistrationController
I0515 17:50:55.753707       1 nonstructuralschema_controller.go:204] Shutting down NonStructuralSchemaConditionController
I0515 17:50:55.753758       1 establishing_controller.go:87] Shutting down EstablishingController
I0515 17:50:55.753796       1 naming_controller.go:302] Shutting down NamingConditionController
I0515 17:50:55.753822       1 apiapproval_controller.go:198] Shutting down KubernetesAPIApprovalPolicyConformantConditionController
I0515 17:50:55.753852       1 crd_finalizer.go:278] Shutting down CRDFinalizer
I0515 17:50:55.753887       1 customresource_discovery_controller.go:245] Shutting down DiscoveryController
I0515 17:50:55.753915       1 available_controller.go:503] Shutting down AvailableConditionController
I0515 17:50:55.754025       1 controller.go:89] Shutting down OpenAPI AggregationController
I0515 17:50:55.754046       1 controller.go:86] Shutting down OpenAPI V3 AggregationController
I0515 17:50:55.754081       1 dynamic_serving_content.go:146] "Shutting down controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key"
I0515 17:50:55.754134       1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting"
I0515 17:50:55.754245       1 dynamic_cafile_content.go:171] "Shutting down controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:50:55.754525       1 cluster_authentication_trust_controller.go:463] Shutting down cluster_authentication_trust_controller controller
I0515 17:50:55.754597       1 dynamic_cafile_content.go:171] "Shutting down controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt"
I0515 17:50:55.755394       1 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:50:55.755539       1 tlsconfig.go:255] "Shutting down DynamicServingCertificateController"
I0515 17:50:55.755493       1 secure_serving.go:255] Stopped listening on [::]:6443
I0515 17:50:55.755742       1 dynamic_serving_content.go:146] "Shutting down controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key"
I0515 17:50:55.754245       1 dynamic_cafile_content.go:171] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
I0515 17:50:56.202330       1 trace.go:205] Trace[287869255]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:53.192) (total time: 3009ms):
Trace[287869255]: [3.009431743s] [3.009431743s] END
W0515 17:50:56.202405       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:50:56.202425       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:00.207279       1 trace.go:205] Trace[1949987037]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:50:57.202) (total time: 3004ms):
Trace[1949987037]: [3.004401066s] [3.004401066s] END
W0515 17:51:00.207355       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:00.207379       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:04.213154       1 trace.go:205] Trace[1488463345]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:01.208) (total time: 3004ms):
Trace[1488463345]: [3.004534501s] [3.004534501s] END
W0515 17:51:04.213229       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:04.213248       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:08.218396       1 trace.go:205] Trace[996780107]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:05.214) (total time: 3004ms):
Trace[996780107]: [3.00417563s] [3.00417563s] END
W0515 17:51:08.218468       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:08.218815       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:12.225289       1 trace.go:205] Trace[1359912236]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:09.219) (total time: 3005ms):
Trace[1359912236]: [3.005966479s] [3.005966479s] END
W0515 17:51:12.225726       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:12.225789       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:16.233065       1 trace.go:205] Trace[1939012164]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:13.227) (total time: 3005ms):
Trace[1939012164]: [3.005906292s] [3.005906292s] END
W0515 17:51:16.233189       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:16.233221       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:20.238277       1 trace.go:205] Trace[582931785]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:17.233) (total time: 3004ms):
Trace[582931785]: [3.004471313s] [3.004471313s] END
W0515 17:51:20.238355       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:20.238382       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...
I0515 17:51:24.248342       1 trace.go:205] Trace[2105727503]: "List(recursive=true) etcd3" audit-id:,key:/secrets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (15-May-2023 17:51:21.238) (total time: 3009ms):
Trace[2105727503]: [3.009664569s] [3.009664569s] END
W0515 17:51:24.248418       1 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"
E0515 17:51:24.248439       1 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/default/secret-post-deploy": failed get version from remote KMS provider: rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "transport: Error while dialing dial unix /opt/vault-kms/vaultkms.socket: connect: no such file or directory"; reinitializing...