Fresh kubernetes deployment using docker runtime - Unable to start api-server and kubelet

[SHeisnam]

Description

OS: Ubuntu 20.04.6 LTS Kuberentes: kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"} -->

Describe the results you received and expected

kubeadm init --pod-network-cidr=16.10.0.0/8 --apiserver-advertise-address=16.32.12.161 --ignore-preflight-errors=NumCPU,Mem --v=6 I0113 04:33:40.032528 4844 initconfiguration.go:117] detected and using CRI socket: unix:///var/run/containerd/containerd.sock I0113 04:33:40.032686 4844 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd" I0113 04:33:40.035595 4844 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt I0113 04:33:42.245729 4844 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28 I0113 04:33:42.245807 4844 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.28.txt [init] Using Kubernetes version: v1.28.5 [preflight] Running pre-flight checks I0113 04:33:44.898098 4844 checks.go:563] validating Kubernetes and kubeadm version I0113 04:33:44.898139 4844 checks.go:168] validating if the firewall is enabled and active I0113 04:33:44.914418 4844 checks.go:203] validating availability of port 6443 I0113 04:33:44.914979 4844 checks.go:203] validating availability of port 10259 I0113 04:33:44.915330 4844 checks.go:203] validating availability of port 10257 I0113 04:33:44.915669 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml I0113 04:33:44.915821 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml I0113 04:33:44.915987 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml I0113 04:33:44.916143 4844 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml I0113 04:33:44.916351 4844 checks.go:430] validating if the connectivity type is via proxy or direct I0113 04:33:44.916407 4844 checks.go:469] validating http connectivity to first IP address in the CIDR I0113 04:33:44.916449 4844 checks.go:469] validating http connectivity to first IP address in the CIDR I0113 04:33:44.916477 4844 checks.go:104] validating the container runtime I0113 04:33:44.954410 4844 checks.go:639] validating whether swap is enabled or not I0113 04:33:44.954552 4844 checks.go:370] validating the presence of executable crictl I0113 04:33:44.954621 4844 checks.go:370] validating the presence of executable conntrack I0113 04:33:44.954671 4844 checks.go:370] validating the presence of executable ip I0113 04:33:44.954722 4844 checks.go:370] validating the presence of executable iptables I0113 04:33:44.954778 4844 checks.go:370] validating the presence of executable mount I0113 04:33:44.954822 4844 checks.go:370] validating the presence of executable nsenter I0113 04:33:44.954867 4844 checks.go:370] validating the presence of executable ebtables I0113 04:33:44.954913 4844 checks.go:370] validating the presence of executable ethtool I0113 04:33:44.954956 4844 checks.go:370] validating the presence of executable socat I0113 04:33:44.954999 4844 checks.go:370] validating the presence of executable tc I0113 04:33:44.955040 4844 checks.go:370] validating the presence of executable touch I0113 04:33:44.955084 4844 checks.go:516] running all checks I0113 04:33:44.967646 4844 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost I0113 04:33:44.967745 4844 checks.go:605] validating kubelet version I0113 04:33:45.000022 4844 checks.go:130] validating if the "kubelet" service is enabled and active I0113 04:33:45.006196 4844 checks.go:203] validating availability of port 10250 I0113 04:33:45.006243 4844 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables I0113 04:33:45.006267 4844 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward I0113 04:33:45.006277 4844 checks.go:203] validating availability of port 2379 I0113 04:33:45.006289 4844 checks.go:203] validating availability of port 2380 I0113 04:33:45.006302 4844 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' I0113 04:33:45.006374 4844 checks.go:828] using image pull policy: IfNotPresent I0113 04:33:45.023755 4844 checks.go:854] pulling: registry.k8s.io/kube-apiserver:v1.28.5 I0113 04:34:53.454374 4844 checks.go:854] pulling: registry.k8s.io/kube-controller-manager:v1.28.5 I0113 04:35:46.053991 4844 checks.go:854] pulling: registry.k8s.io/kube-scheduler:v1.28.5 I0113 04:37:02.823926 4844 checks.go:854] pulling: registry.k8s.io/kube-proxy:v1.28.5 W0113 04:37:24.696705 4844 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image. I0113 04:37:24.713794 4844 checks.go:854] pulling: registry.k8s.io/pause:3.9 I0113 04:37:30.001247 4844 checks.go:854] pulling: registry.k8s.io/etcd:3.5.9-0 I0113 04:38:51.184189 4844 checks.go:854] pulling: registry.k8s.io/coredns/coredns:v1.10.1 [certs] Using certificateDir folder "/etc/kubernetes/pki" I0113 04:39:33.027558 4844 certs.go:112] creating a new certificate authority for ca [certs] Generating "ca" certificate and key I0113 04:39:33.122552 4844 certs.go:519] validating certificate period for ca certificate [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 16.32.12.161] [certs] Generating "apiserver-kubelet-client" certificate and key I0113 04:39:33.314136 4844 certs.go:112] creating a new certificate authority for front-proxy-ca [certs] Generating "front-proxy-ca" certificate and key I0113 04:39:33.392288 4844 certs.go:519] validating certificate period for front-proxy-ca certificate [certs] Generating "front-proxy-client" certificate and key I0113 04:39:33.699882 4844 certs.go:112] creating a new certificate authority for etcd-ca [certs] Generating "etcd/ca" certificate and key I0113 04:39:33.884850 4844 certs.go:519] validating certificate period for etcd/ca certificate [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com localhost] and IPs [16.32.12.161 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [cvs-k8s-master1.hpemf.com localhost] and IPs [16.32.12.161 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key I0113 04:39:34.527846 4844 certs.go:78] creating new public/private key files for signing service account users [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" I0113 04:39:34.678520 4844 kubeconfig.go:103] creating kubeconfig file for admin.conf [kubeconfig] Writing "admin.conf" kubeconfig file I0113 04:39:34.781723 4844 kubeconfig.go:103] creating kubeconfig file for kubelet.conf [kubeconfig] Writing "kubelet.conf" kubeconfig file I0113 04:39:34.871489 4844 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf [kubeconfig] Writing "controller-manager.conf" kubeconfig file I0113 04:39:34.967682 4844 kubeconfig.go:103] creating kubeconfig file for scheduler.conf [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" I0113 04:39:35.110542 4844 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" I0113 04:39:35.110632 4844 manifests.go:102] [control-plane] getting StaticPodSpecs I0113 04:39:35.110757 4844 certs.go:519] validating certificate period for CA certificate I0113 04:39:35.110810 4844 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-apiserver" I0113 04:39:35.110836 4844 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver" I0113 04:39:35.110856 4844 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-apiserver" I0113 04:39:35.110876 4844 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-apiserver" I0113 04:39:35.110898 4844 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver" I0113 04:39:35.110919 4844 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver" I0113 04:39:35.111328 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml" [control-plane] Creating static Pod manifest for "kube-controller-manager" I0113 04:39:35.111378 4844 manifests.go:102] [control-plane] getting StaticPodSpecs I0113 04:39:35.111487 4844 manifests.go:128] [control-plane] adding volume "ca-certs" for component "kube-controller-manager" I0113 04:39:35.111512 4844 manifests.go:128] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager" I0113 04:39:35.111532 4844 manifests.go:128] [control-plane] adding volume "etc-pki" for component "kube-controller-manager" I0113 04:39:35.111553 4844 manifests.go:128] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager" I0113 04:39:35.111573 4844 manifests.go:128] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager" I0113 04:39:35.111593 4844 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager" I0113 04:39:35.111613 4844 manifests.go:128] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager" I0113 04:39:35.111633 4844 manifests.go:128] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager" I0113 04:39:35.112014 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [control-plane] Creating static Pod manifest for "kube-scheduler" I0113 04:39:35.112064 4844 manifests.go:102] [control-plane] getting StaticPodSpecs I0113 04:39:35.112162 4844 manifests.go:128] [control-plane] adding volume "kubeconfig" for component "kube-scheduler" I0113 04:39:35.113257 4844 manifests.go:157] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml" I0113 04:39:35.113276 4844 kubelet.go:67] Stopping the kubelet [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet I0113 04:39:35.435924 4844 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy I0113 04:39:35.436329 4844 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s I0113 04:39:45.445701 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10008 milliseconds I0113 04:39:55.947643 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:40:06.447590 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds [kubelet-check] Initial timeout of 40s passed. I0113 04:40:16.947274 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:40:27.449973 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:40:37.948672 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds I0113 04:40:48.454894 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:40:58.947357 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:41:09.447811 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds I0113 04:41:19.948021 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:41:24.015319 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 3569 milliseconds I0113 04:41:34.448399 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds

I0113 04:41:44.947303 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10001 milliseconds I0113 04:41:55.047823 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10100 milliseconds I0113 04:42:05.048792 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:42:15.447846 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:42:25.953775 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10006 milliseconds I0113 04:42:35.958271 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10004 milliseconds I0113 04:42:45.044447 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 8592 milliseconds I0113 04:42:55.454360 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10007 milliseconds I0113 04:43:05.947112 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:43:06.466243 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 518 milliseconds I0113 04:43:16.966248 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:43:27.446466 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:43:27.978857 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 532 milliseconds I0113 04:43:38.447348 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10000 milliseconds I0113 04:43:48.948740 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10002 milliseconds I0113 04:43:58.957933 4844 round_trippers.go:553] GET https://16.32.12.161:6443/healthz?timeout=10s in 10008 milliseconds

Unfortunately, an error has occurred: timed out waiting for the condition

This error is likely caused by:

• The kubelet is not running
• The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:

• 'systemctl status kubelet'
• 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl:

• 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with:
• 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID' couldn't initialize a Kubernetes cluster k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:108 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1 cmd/kubeadm/app/cmd/phases/workflow/runner.go:259 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll cmd/kubeadm/app/cmd/phases/workflow/runner.go:446 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run cmd/kubeadm/app/cmd/phases/workflow/runner.go:232 k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1 cmd/kubeadm/app/cmd/init.go:111 github.com/spf13/cobra.(Command).execute vendor/github.com/spf13/cobra/command.go:940 github.com/spf13/cobra.(Command).ExecuteC vendor/github.com/spf13/cobra/command.go:1068 github.com/spf13/cobra.(Command).Execute vendor/github.com/spf13/cobra/command.go:992 k8s.io/kubernetes/cmd/kubeadm/app.Run cmd/kubeadm/app/kubeadm.go:50 main.main cmd/kubeadm/kubeadm.go:25 runtime.main /usr/local/go/src/runtime/proc.go:250 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 error execution phase wait-control-plane k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run.func1 cmd/kubeadm/app/cmd/phases/workflow/runner.go:260 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).visitAll cmd/kubeadm/app/cmd/phases/workflow/runner.go:446 k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(Runner).Run cmd/kubeadm/app/cmd/phases/workflow/runner.go:232 k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1 cmd/kubeadm/app/cmd/init.go:111 github.com/spf13/cobra.(Command).execute vendor/github.com/spf13/cobra/command.go:940 github.com/spf13/cobra.(Command).ExecuteC vendor/github.com/spf13/cobra/command.go:1068 github.com/spf13/cobra.(Command).Execute vendor/github.com/spf13/cobra/command.go:992 k8s.io/kubernetes/cmd/kubeadm/app.Run cmd/kubeadm/app/kubeadm.go:50 main.main cmd/kubeadm/kubeadm.go:25 runtime.main /usr/local/go/src/runtime/proc.go:250 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause f09fd6b7be496 9ecc4287300e3 About a minute ago Exited kube-apiserver 4 7766ed4728e56 kube-apiserver-cvs-k8s-master1.hpemf.com 54895a4902925 babc03668f18a 4 minutes ago Running kube-scheduler 0 22bfa846906f3 kube-scheduler-cvs-k8s-master1.hpemf.com d61e404e4c8ee c527ad14e0cd5 4 minutes ago Running kube-controller-manager 0 b4c9d707087be kube-controller-manager-cvs-k8s-master1.hpemf.com cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ crictl logs f09fd6b7be496 WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the defaul t settings are now deprecated, you should set the endpoint instead. E0113 04:44:30.250083 6325 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" containerID="f09fd6b7be496" FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ crictl logs f09fd6b7be496 -p WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the defaul t settings are now deprecated, you should set the endpoint instead. E0113 04:44:48.366198 6381 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" containerID="f09fd6b7be496" FATA[0000] rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID E0113 04:45:21.658814 6504 remote_runtime.go:415] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"CONTAINERID\": ot found" containerID="CONTAINERID" FATA[0000] rpc error: code = NotFound desc = an error occurred when try to find container "CONTAINERID": not found cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ journalctl -xeu kubelet Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:45:58.806025 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list v1.Service: Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:45:58.806618 5121 trace.go:236] Trace[1316285868]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:150 (13-Jan-2024 04:45:28.805) (total time: 30001ms): Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[1316285868]: ---"Objects listed" error:Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout 30000ms (04:45:58.806) Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[1316285868]: [30.001489569s] [30.001489569s] END Jan 13 04:45:58 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:45:58.807015 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch v1.Service: failed to list v1.Service: Get "https://16.32.12.161:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout Jan 13 04:45:59 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:45:59.568418 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" interval="7s" Jan 13 04:46:06 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:06.426353 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"cvs-k8s-master1.hpemf.com\" not found" Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.015055 5121 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSecond> Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:08.018895 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee" Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.019047 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:08 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:08.019821 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)\"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b" Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:10.018782 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375" Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:10.018938 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:10 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:10.019463 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)\"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279" Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794748 5121 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://16.32.12.161:6443/api/v1/nodes\": dial tcp 16.32.12.161:6443: connect: no route to host" node="cvs-k8s-master1.hpemf.com" Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794757 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": dial tcp 16.32.12.161:6443: connect: no route to host" interval="7s" Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.794920 5121 certificate_manager.go:562] kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post "https://16.32.12.161:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 16.32.12.161:6443: connect: no route to host Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:46:13.796329 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list v1.RuntimeClass: Get "https://16.32.12.161:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: connect: no route to host Jan 13 04:46:13 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:13.796492 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch v1.RuntimeClass: failed to list v1.RuntimeClass: Get "https://16.32.12.161:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: connect: no route to host Jan 13 04:46:16 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:16.426807 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"cvs-k8s-master1.hpemf.com\" not found" Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:19.021144 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee" Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:19.022454 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:19 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:19.024282 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)\"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b" Jan 13 04:46:20 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:20.802260 5121 kubelet_node_status.go:70] "Attempting to register node" node="cvs-k8s-master1.hpemf.com" Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:23.019339 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375" Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:23.019554 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:23 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:23.020038 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)\"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279" Jan 13 04:46:26 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:26.427263 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"cvs-k8s-master1.hpemf.com\" not found" Jan 13 04:46:30 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:30.800939 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": dial tcp 16.32.12.161:6443: i/o timeout" interval="7s" Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:31.019007 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee" Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.019153 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.019949 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)\"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b" Jan 13 04:46:31 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:31.021470 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:36 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:36.429360 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"cvs-k8s-master1.hpemf.com\" not found" Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:37.019963 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375" Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:37.020194 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:37 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:37.022300 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)\"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279" Jan 13 04:46:39 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:39.019158 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:45.019159 5121 scope.go:117] "RemoveContainer" containerID="b901677470f98e7867473dd75677c055110a4fab8f523fd027fbb64fed5e70ee" Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:45.019289 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:45 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:45.021634 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-cvs-k8s-master1.hpemf.com_kube-system(5a9281867058526c845d9c79d7ac566b)\"" pod="kube-system/kube-apiserver-cvs-k8s-master1.hpemf.com" podUID="5a9281867058526c845d9c79d7ac566b" Jan 13 04:46:46 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:46.430534 5121 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"cvs-k8s-master1.hpemf.com\" not found" Jan 13 04:46:47 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:47.803608 5121 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://16.32.12.161:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/cvs-k8s-master1.hpemf.com?timeout=10s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" interval="7s" Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.017029 5121 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSecond> Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.017228 5121 event.go:228] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cvs-k8s-master1.hpemf.com.17a9ce4fc6aa2e27", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds> Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:48.019835 5121 scope.go:117] "RemoveContainer" containerID="8918b8916435863ba9c157f34992778454f92114e3456623b09d598006eba375" Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.020044 5121 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.1.1 8.8.8.8 1.1.1.1" Jan 13 04:46:48 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:48.020590 5121 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"etcd\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=etcd pod=etcd-cvs-k8s-master1.hpemf.com_kube-system(c4c04f0bb3e857970e9d48c8f41ae279)\"" pod="kube-system/etcd-cvs-k8s-master1.hpemf.com" podUID="c4c04f0bb3e857970e9d48c8f41ae279" Jan 13 04:46:50 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:50.803379 5121 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://16.32.12.161:6443/api/v1/nodes\": dial tcp 16.32.12.161:6443: i/o timeout" node="cvs-k8s-master1.hpemf.com" Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: W0113 04:46:55.047513 5121 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list v1.CSIDriver: Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: I0113 04:46:55.047672 5121 trace.go:236] Trace[581891402]: "Reflector ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:150 (13-Jan-2024 04:46:25.046) (total time: 30000ms): Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[581891402]: ---"Objects listed" error:Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout 30000ms (04:46:55.047) Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: Trace[581891402]: [30.000786886s] [30.000786886s] END Jan 13 04:46:55 cvs-k8s-master1.hpemf.com kubelet[5121]: E0113 04:46:55.047699 5121 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://16.32.12.161:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 16.32.12.161:6443: i/o timeout lines 2844-2897/2897 (END)

Has anyone face such problem

What version of containerd are you using?

containerd --version containerd containerd.io 1.6.26 3dd1e886e55dd695541fdcd67420c2888645a495

Any other relevant information

cvs-k8s-master1 $ crictl info WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. E0113 05:19:17.843289 7784 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" FATA[0000] getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ cvs-k8s-master1 $ uname -a Linux cvs-k8s-master1.hpemf.com 5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux cvs-k8s-master1 $

Show configuration if it is related to CRI plugin.

mv /etc/containerd/config.toml /etc/containerd/config.toml_old sudo systemctl restart containerd sudo systemctl status containerd

cvs-k8s-master1 $ cat /etc/containerd/config.toml_old

Copyright 2018-2022 Docker Inc.

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

disabled_plugins = ["cri"]

root = "/var/lib/containerd"

state = "/run/containerd"

subreaper = true

oom_score = 0

[grpc]

address = "/run/containerd/containerd.sock"

uid = 0

gid = 0

[debug]

address = "/run/containerd/debug.sock"

uid = 0

gid = 0

level = "info"

[liangyuanpeng]

@h-182 I think you need to report it to https://github.com/Mirantis/cri-dockerd or kubernetes, and not containerd.

[rameshkothamasu]

I came across this statement: https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/#cri-api-removal Kubernetes v1.26 will not support CRI v1alpha2. That removal will result in the kubelet not registering the node if the container runtime doesn't support CRI v1. This means that containerd minor version 1.5 and older will not be supported in Kubernetes 1.26; if you use containerd, you will need to upgrade to containerd version 1.6.0 or later before you upgrade that node to Kubernetes v1.26. Other container runtimes that only support the v1alpha2 are equally affected: if that affects you, you should contact the container runtime vendor for advice or check their website for additional instructions in how to move forward.