Control plane static pod does not generate due to race condition

Gentoli commented 1 year ago

Bug Report

Description

If a user provided static pod mounts /system/secrets/kubernetes/kube-controller-manager/kubeconfig with out explicit type, control plane pods does not gets created due to

192.168.2.205: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.RenderSecretsStaticPodController", "error": "error writing template \"kubeconfig\" for \"kube-controller-manager\": open /system/secrets/kubernetes/kube-controller-manager/kubeconfig: is a directory"}

I think is caused by the user provided static pods being supplied to kubelet before the control plane pods.

So the steps goes like this:

on start, static pod endpoint contains only user supplied pods
kubelet tries to create the pod with mount /system/secrets/kubernetes/kube-controller-manager/kubeconfig
- file does not exist, containerd create a folder in it's place
machined tries to create resources for control plane pods
- /system/secrets/kubernetes/kube-controller-manager/kubeconfig exists as a folder, pods does not get generated

To protect the control plane startup, either

the operation for providing static pods (at least on node startup) should be atomic (either all static pods get provided at once or none) due to the resource generating setup from RenderSecretsStaticPodController.
the resources for control plane pods can be generated before the static pods setup.

The workaround is to use type=Directory on the parent so the file is not created (also the kubeconfig is not visible to kubelet).

 - hostPath:
     path: /system/secrets/kubernetes/kube-controller-manager
     type: Directory

Logs

machined failed to write kubeconfig

192.168.2.205: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.RenderSecretsStaticPodController", "error": "error writing template \"kubeconfig\" for \"kube-controller-manager\": open /system/secrets/kubernetes/kube-controller-manager/kubeconfig: is a directory"}

endpoint with no pods

talos-xmd-rtx:~# curl http://127.0.0.1:46817
kind: PodList
items: []

Environment

Talos version: [talosctl version --nodes <problematic nodes>]

Client:
      Tag:         v1.3.5
      SHA:         03edf8c1
      Built:       
      Go version:  go1.19.6
      OS/Arch:     darwin/amd64
Server:
      NODE:        192.168.2.205
      Tag:         v1.3.5
      SHA:         03edf8c1
      Built:       
      Go version:  go1.19.6
      OS/Arch:     linux/amd64
      Enabled:     RBAC

Kubernetes version: [kubectl version --short] N/A (kube-apiservier not running)
Platform: KubeVirt / OKD

smira commented 1 year ago

yes, this is expected behavior - in general, not the best idea to re-use kubeconfig from other Talos static pods, as things might change in the future.

github-actions[bot] commented 6 days ago

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 1 day ago

This issue was closed because it has been stalled for 7 days with no activity.

siderolabs / talos