timothysc
commented
6 years ago /assign @chuckha
Closed ghoben closed 6 years ago
timothysc
commented
6 years ago /assign @chuckha
kad
commented
6 years ago Based on below:
May 05 12:19:32 gernot-pi3 kubelet[11521]: E0505 12:19:32.267865 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://**192.168.1.123:6443**/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
Do you have something on your system that might prevent connection to 192.168.1.123:6443 ? It might be firewall rules, or incorrect detection of primary IP address for master.
ghoben
commented
6 years ago I have not setup something on purpose, but was suprised that iptable showed something. Not sure if that is something which might block.
I checked with iptables -L
$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT) target prot opt source destination KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references) target prot opt source destination
Chain DOCKER-ISOLATION (1 references) target prot opt source destination RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all -- anywhere anywhere / kubernetes firewall for dropping marked packets / mark match 0x8000/0x8000
ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:2a:fa:df:4d txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.123 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::18a2:282b:94b5:af9e prefixlen 64 scopeid 0x20 ether b8:27:eb:41:5f:51 txqueuelen 1000 (Ethernet) RX packets 30137 bytes 1806994 (1.7 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2652 bytes 237763 (232.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
wlan0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether b8:27:eb:14:0a:04 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I don't see something which may cause a blocking.
ghoben
commented
6 years ago Looking again at the logs I see the read out of the CPU L2 cache does not work. Is that really an Error (@kad ) and prevents something, or is this only a warning. I checked manually that file really does not exists for read out.
May 05 12:18:33 gernot-pi3 kubelet[11521]: E0505 12:18:33.997223 11521 machine.go:194] failed to >get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory
kad
commented
6 years ago your FORWARD policy in firewall set to DROP. This might be the reason. Try to disable system firewall scripts and retry.
ghoben
commented
6 years ago I changed the default of FORWARD: sudo iptables -P FORWARD ACCEPT. I also checked the count for this which was always 0. I flushed the iptables. As soon as the kubelet.service starts up it registered the rule for dropping marked packages. The problem still exists.
kad
commented
6 years ago if you do manually telnet 192.168.1.123 6443 which error message you get ?
kad
commented
6 years ago Also, during kubeadm waiting period, do you see in docker ps or docker images that images of control plane is getting pulled and some container started ? it might be because of combination of slow network + slow hardware, timeout in kubeadm is not big enough.
ghoben
commented
6 years ago When I telnet on the same box, I get a connect.
telnet 192.168.1.123 6443 Trying 192.168.1.123... Connected to 192.168.1.123. Escape character is '^]'.
docker containers are started and stopped: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9aa3b2dd07b4 c68f5521f86b "kube-apiserver --se…" About a minute ago Up About a minute k8s_kube-apiserver_kube-apiserver-gernot-pi3_kube-system_6761abf6cea65 52c7b386552482c777c_81 f58d3b777822 88c32b5960ff "etcd --advertise-cl…" About a minute ago Up About a minute k8s_etcd_etcd-gernot-pi3_kube-system16846a2efbc5ef643330dac361a49ad0 75 ef29701c46a9 f67c023adb1b "kube-controller-man…" 6 hours ago Up 6 hours k8s_kube-controller-manager_kube-controller-manager-gernot-pi3_kube-sy stem_98578eb479695566b995bf6ae33a05fe_0 2f25273d6609 816c40ff51c0 "kube-scheduler --ku…" 6 hours ago Up 6 hours k8s_kube-scheduler_kube-scheduler-gernot-pi3_kube-system_75b1b9ab0f3a3 7601ee6e0a6c14cc1a7_0 3efb61660a92 k8s.gcr.io/pause-arm:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_kube-controller-manager-gernot-pi3_kube-system_98578eb47969556 6b995bf6ae33a05fe_0 87d29b2c1317 k8s.gcr.io/pause-arm:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_kube-apiserver-gernot-pi3_kube-system_6761abf6cea6552c7b386552 482c777c_0 d6359c887faf k8s.gcr.io/pause-arm:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_etcd-gernot-pi3_kube-system_16846a2efbc5ef643330dac361a49ad0_0 bc333865d025 k8s.gcr.io/pause-arm:3.1 "/pause" 6 hours ago Up 6 hours k8s_POD_kube-scheduler-gernot-pi3_kube-system_75b1b9ab0f3a37601ee6e0a6 c14cc1a7_0
etcd and kube-apiserver are alive for about 2 min an then they are stopped. After awhile new containers are started up.
ghoben
commented
6 years ago Here the log of the api-server, after it got stopped.
Flag --insecure-port has been deprecated, This flag will be removed in a future version. Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version. I0511 18:00:16.556559 1 server.go:135] Version: v1.10.2 I0511 18:00:16.557712 1 server.go:724] external host was not specified, using 192.168.1.123 I0511 18:00:41.563008 1 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota. I0511 18:00:41.571854 1 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota. I0511 18:00:41.619309 1 master.go:228] Using reconciler: master-count W0511 18:00:42.739861 1 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources. W0511 18:00:42.888160 1 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0511 18:00:42.900681 1 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0511 18:00:43.006522 1 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. [restful] 2018/05/11 18:00:43 log.go:33: [restful/swagger] listing is available at https://192.168.1.123:6443/swaggerapi [restful] 2018/05/11 18:00:43 log.go:33: [restful/swagger] https://192.168.1.123:6443/swaggerui/ is mapped to folder /swagger-ui/ [restful] 2018/05/11 18:01:08 log.go:33: [restful/swagger] listing is available at https://192.168.1.123:6443/swaggerapi [restful] 2018/05/11 18:01:08 log.go:33: [restful/swagger] https://192.168.1.123:6443/swaggerui/ is mapped to folder /swagger-ui/ I0511 18:01:08.937471 1 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota. I0511 18:01:57.560880 1 serve.go:96] Serving securely on [::]:6443 I0511 18:01:57.562592 1 controller.go:84] Starting OpenAPI AggregationController I0511 18:01:57.562697 1 controller.go:90] Shutting down OpenAPI AggregationController I0511 18:01:57.569815 1 serve.go:136] Stopped listening on [::]:6443 I0511 18:01:57.581678 1 crd_finalizer.go:242] Starting CRDFinalizer I0511 18:01:57.581874 1 crd_finalizer.go:246] Shutting down CRDFinalizer I0511 18:01:57.586556 1 apiservice_controller.go:90] Starting APIServiceRegistrationController I0511 18:01:57.586665 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller E0511 18:01:57.586800 1 cache.go:35] Unable to sync caches for APIServiceRegistrationController controller I0511 18:01:57.586879 1 apiservice_controller.go:94] Shutting down APIServiceRegistrationController I0511 18:01:57.589276 1 available_controller.go:262] Starting AvailableConditionController I0511 18:01:57.589372 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller E0511 18:01:57.589504 1 cache.go:35] Unable to sync caches for AvailableConditionController controller I0511 18:01:57.589586 1 available_controller.go:266] Shutting down AvailableConditionController I0511 18:01:57.601141 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40312: tls: first record does not look like a TLS handshake I0511 18:01:57.662069 1 crdregistration_controller.go:110] Starting crd-autoregister controller I0511 18:01:57.662235 1 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller E0511 18:01:57.662388 1 controller_utils.go:1022] Unable to sync caches for crd-autoregister controller I0511 18:01:57.662472 1 crdregistration_controller.go:115] Shutting down crd-autoregister controller I0511 18:01:57.675788 1 customresource_discovery_controller.go:174] Starting DiscoveryController E0511 18:01:57.675984 1 customresource_discovery_controller.go:177] timed out waiting for caches to sync I0511 18:01:57.676067 1 customresource_discovery_controller.go:178] Shutting down DiscoveryController I0511 18:01:57.676166 1 naming_controller.go:276] Starting NamingConditionController I0511 18:01:57.676261 1 naming_controller.go:280] Shutting down NamingConditionController I0511 18:01:57.839210 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40320: EOF I0511 18:01:57.857039 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40316: EOF I0511 18:01:57.860771 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40310: EOF I0511 18:01:57.873825 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40314: EOF I0511 18:01:57.890494 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40322: EOF I0511 18:01:57.900313 1 logs.go:49] http: TLS handshake error from 192.168.1.123:40318: EOF E0511 18:01:58.570621 1 storage_rbac.go:157] unable to initialize clusterroles: Get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles: dial tcp 127.0.0.1:6443: getsockopt: connection refused
ghoben
commented
6 years ago E0511 18:01:57.586800 1 cache.go:35] Unable to sync caches for APIServiceRegistrationController controller
does this mean the api-server has problems connecting to the etcd? Or which caches are synced here.
ghoben
commented
6 years ago @kad : I finally figured out the root cause. Since I retried to work with kubeadm init several times, kubelet had some old persistent data which was not cleared on startup within /var/lib/kubelet/pki. I ended up whipping the /var/lib/kubelet directory. After restarting kubelet the TLS problems where gone and kubelet was able to connect to the etcd and api-server. Not sure why on the start the PEM files within the /var/lib/kubelet are not recreated if the kubelet service has other defaults. I would suggest adding this as warning to the kubeadm that there has been an kubelet service already running partly. I also had to downgrade for docker to docker-ce:18.04-ce, since docker-ce.18.05 was only creating the containers, but never started them up. The containers (kube-apiserver, etcd, scheduler, controller) stayed at "Created" for ever. Thanks for your help.
ghoben
commented
6 years ago For others which may experience something like the same. Try to alter the livenessProbe setting of the /etc/kubernetes/manifests/kube-apiserver.yaml. Increase the initialDelaySeconds: 60. this helps also for slow devices. I filled a feature request that this should be considered in one way or the other. https://github.com/kubernetes/kubeadm/issues/931
What keywords did you search in kubeadm issues before filing this one?
If you have found any duplicates, you should instead reply there and close this page.
kubeadm init is hanging for a lot of reasons in the forum. I tried hints of other threads: aligning the cgroup_memory settings, turned of the swap, static IP. But it must be something else. The CNI warning, I was told can be ignored, since setting up Flannel will only work of the master node initializes first.
Is this a BUG REPORT or FEATURE REQUEST?
Choose one: BUG REPORT or FEATURE REQUEST BUG REPORT
Versions
kubeadm version (use
kubeadm version):Environment:
Kubernetes version (use
kubectl version): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}Cloud provider or hardware configuration: Raspberry Pi3
OS (e.g. from /etc/os-release): PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)" NAME="Raspbian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs" HYPRIOT_OS="HypriotOS/armhf" HYPRIOT_OS_VERSION="v2.0.1" HYPRIOT_DEVICE="Raspberry Pi" HYPRIOT_IMAGE_VERSION="v1.9.0"
Kernel (e.g.
uname -a): Linux gernot-pi3 4.14.34-hypriotos-v7+ #1 SMP Sun Apr 22 14:57:31 UTC 2018 armv7l GNU/LinuxOthers:
What happened?
I wanted to create a simple master node on a raspberry PI3. (following instructions: https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/. Docker is running successfully, installed latest kubeadm v10.2 packages. The initialization of the kubeadm locks up in an infinite loop. The docker containers etcd and kubeapiserver are restarting after 2min.
What you expected to happen?
That the kubeadm init succeeds
How to reproduce it (as minimally and precisely as possible)?
take a raspberry PI3 with stretch, apt-get install kubeadm and call kubeadm init --pod-network-cidr 10.244.0.0/16 . I tried it with plain kubeadm init and also with raspbian latest distribution nothing worked.
Anything else we need to know?
May 05 12:18:32 gernot-pi3 systemd[1]: Started kubelet: The Kubernetes Node Agent. -- Subject: Unit kubelet.service has finished start-up -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- Unit kubelet.service has finished starting up. -- -- The start-up result is done. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --allow-privileged has been deprecated, will be removed in a future version May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 05 12:18:33 gernot-pi3 kubelet[11521]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.12, and the cadvisor port will be removed entirely in 1.13 May 05 12:18:33 gernot-pi3 kubelet[11521]: I0505 12:18:33.641815 11521 feature_gate.go:226] feature gates: &{{} map[]} May 05 12:18:33 gernot-pi3 kubelet[11521]: W0505 12:18:33.702441 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d May 05 12:18:33 gernot-pi3 kubelet[11521]: W0505 12:18:33.735387 11521 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup. May 05 12:18:33 gernot-pi3 kubelet[11521]: I0505 12:18:33.735661 11521 server.go:376] Version: v1.10.2 May 05 12:18:33 gernot-pi3 kubelet[11521]: I0505 12:18:33.736024 11521 feature_gate.go:226] feature gates: &{{} map[]} May 05 12:18:33 gernot-pi3 kubelet[11521]: I0505 12:18:33.736768 11521 plugins.go:89] No cloud provider specified. May 05 12:18:33 gernot-pi3 kubelet[11521]: I0505 12:18:33.773265 11521 certificate_store.go:117] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem". May 05 12:18:33 gernot-pi3 kubelet[11521]: E0505 12:18:33.997223 11521 machine.go:194] failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.007343 11521 server.go:613] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.009120 11521 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: / May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.009246 11521 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.15} GracePeriod:0s MinReclaim:} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true}
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.010215 11521 container_manager_linux.go:266] Creating device plugin manager: true
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.010469 11521 state_mem.go:36] [cpumanager] initializing new in-memory state store
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.011874 11521 state_mem.go:84] [cpumanager] updated default cpuset: ""
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.012014 11521 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.012775 11521 kubelet.go:272] Adding pod path: /etc/kubernetes/manifests
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.012957 11521 kubelet.go:297] Watching apiserver
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.028287 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.031602 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.031756 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.044766 11521 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.044979 11521 kubelet.go:556] Hairpin mode set to "hairpin-veth"
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.045778 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.045966 11521 client.go:75] Connecting to docker on unix:///var/run/docker.sock
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.046049 11521 client.go:104] Start docker client with request timeout=2m0s
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.066752 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.097991 11521 hostport_manager.go:68] The binary conntrack is not installed, this can cause failures in network connection cleanup.
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.098322 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.098567 11521 docker_service.go:244] Docker cri networking managed by cni
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.177731 11521 docker_service.go:249] Docker Info: &{ID:LNNB:DYWF:DUR6:EDME:PHSM:MSDQ:BFVL:M4JH:DEXQ:IULQ:BMDO:OFBT Containers:10 ContainersRunning:0 ContainersPaused:0 ContainersStopped:10 Images:5 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:21 OomKillDisable:true NGoroutines:34 SystemTime:2018-05-05T12:18:34.107410611Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.14.34-hypriotos-v7+ OperatingSystem:Raspbian GNU/Linux 9 (stretch) OSType:linux Architecture:armv7l IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0x1625ef80 NCPU:4 MemTotal:1024184320 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:gernot-pi3 Labels:[] ExperimentalBuild:false ServerVersion:18.04.0-ce ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:docker-runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:773c489c9c1b21a6d78b5c538cd395416ec50f88 Expected:773c489c9c1b21a6d78b5c538cd395416ec50f88} RuncCommit:{ID:4fc53a81fb7c994640722ac585fa9ca548971871 Expected:4fc53a81fb7c994640722ac585fa9ca548971871} InitCommit:{ID:949e6fa Expected:949e6fa} SecurityOptions:[name=seccomp,profile=default]}
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.178531 11521 docker_service.go:262] Setting cgroupDriver to cgroupfs
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.270395 11521 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.315918 11521 kuberuntime_manager.go:186] Container runtime docker initialized, version: 18.04.0-ce, apiVersion: 1.37.0
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.320597 11521 csi_plugin.go:61] kubernetes.io/csi: plugin initializing...
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.329033 11521 kubelet.go:1277] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.332170 11521 event.go:209] Unable to write event: 'Post https://192.168.1.123:6443/api/v1/namespaces/default/events: dial tcp 192.168.1.123:6443: getsockopt: connection refused' (may retry after sleeping)
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.340796 11521 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.341049 11521 status_manager.go:140] Starting to sync pod status with apiserver
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.341157 11521 kubelet.go:1777] Starting kubelet main sync loop.
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.341379 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.342254 11521 server.go:129] Starting to listen on 0.0.0.0:10250
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.363172 11521 volume_manager.go:247] Starting Kubelet Volume Manager
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.372436 11521 server.go:299] Adding debug handlers to kubelet server.
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.413380 11521 server.go:944] Started kubelet
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.413963 11521 desired_state_of_world_populator.go:129] Desired state populator starts to run
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.451390 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
May 05 12:18:34 gernot-pi3 kubelet[11521]: W0505 12:18:34.458971 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.462124 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.465271 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.538072 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.546598 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.656010 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.747128 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:34 gernot-pi3 kubelet[11521]: I0505 12:18:34.761580 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:34 gernot-pi3 kubelet[11521]: E0505 12:18:34.769129 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:35 gernot-pi3 kubelet[11521]: E0505 12:18:35.030887 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:35 gernot-pi3 kubelet[11521]: E0505 12:18:35.034703 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:35 gernot-pi3 kubelet[11521]: E0505 12:18:35.037421 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:35 gernot-pi3 kubelet[11521]: I0505 12:18:35.056383 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
May 05 12:18:35 gernot-pi3 kubelet[11521]: I0505 12:18:35.169512 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:35 gernot-pi3 kubelet[11521]: I0505 12:18:35.186906 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:35 gernot-pi3 kubelet[11521]: E0505 12:18:35.188799 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:35 gernot-pi3 kubelet[11521]: I0505 12:18:35.856825 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
May 05 12:18:35 gernot-pi3 kubelet[11521]: I0505 12:18:35.989246 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:36 gernot-pi3 kubelet[11521]: I0505 12:18:36.004891 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:36 gernot-pi3 kubelet[11521]: E0505 12:18:36.008624 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:36 gernot-pi3 kubelet[11521]: E0505 12:18:36.032875 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:36 gernot-pi3 kubelet[11521]: E0505 12:18:36.039060 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:36 gernot-pi3 kubelet[11521]: E0505 12:18:36.040452 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:36 gernot-pi3 kubelet[11521]: W0505 12:18:36.610785 11521 nvidia.go:74] Error reading "/sys/bus/pci/devices/": open /sys/bus/pci/devices/: no such file or directory
May 05 12:18:37 gernot-pi3 kubelet[11521]: E0505 12:18:37.046856 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:37 gernot-pi3 kubelet[11521]: E0505 12:18:37.046883 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:37 gernot-pi3 kubelet[11521]: E0505 12:18:37.049291 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.457187 11521 kubelet.go:1794] skipping pod synchronization - [container runtime is down]
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.586624 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.599584 11521 cpu_manager.go:155] [cpumanager] starting with none policy
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.600373 11521 cpu_manager.go:156] [cpumanager] reconciling every 10s
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.600483 11521 policy_none.go:42] [cpumanager] none policy: Start
May 05 12:18:37 gernot-pi3 kubelet[11521]: Starting Device Plugin manager
May 05 12:18:37 gernot-pi3 kubelet[11521]: E0505 12:18:37.608773 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.609623 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:37 gernot-pi3 kubelet[11521]: I0505 12:18:37.622397 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:37 gernot-pi3 kubelet[11521]: E0505 12:18:37.624538 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:38 gernot-pi3 kubelet[11521]: E0505 12:18:38.049608 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:38 gernot-pi3 kubelet[11521]: E0505 12:18:38.053817 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:38 gernot-pi3 kubelet[11521]: E0505 12:18:38.061280 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:39 gernot-pi3 kubelet[11521]: E0505 12:18:39.052474 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:39 gernot-pi3 kubelet[11521]: E0505 12:18:39.056636 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:39 gernot-pi3 kubelet[11521]: E0505 12:18:39.064304 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:39 gernot-pi3 kubelet[11521]: E0505 12:18:39.955535 11521 event.go:209] Unable to write event: 'Post https://192.168.1.123:6443/api/v1/namespaces/default/events: dial tcp 192.168.1.123:6443: getsockopt: connection refused' (may retry after sleeping)
May 05 12:18:40 gernot-pi3 kubelet[11521]: E0505 12:18:40.054540 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: E0505 12:18:40.058709 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: E0505 12:18:40.068191 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.657670 11521 pod_container_deletor.go:77] Container "0e33a546c5c3324b5efb7d90ad60de4a1dd842561a80972c13db1aafa6de51ad" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.657967 11521 pod_container_deletor.go:77] Container "e65102c58996b4e58bb0fa225642cf7435f6ccc472448e4cb2243368b4d287b5" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.658095 11521 pod_container_deletor.go:77] Container "a1aaaed0156452c386c13a14d26aaec4e7f3dfdb74a6142642d52fa2beb9bfcd" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.658714 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.692929 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.693736 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.712327 11521 status_manager.go:461] Failed to get status for pod "etcd-gernot-pi3_kube-system(16846a2efbc5ef643330dac361a49ad0)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/etcd-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.717291 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.717867 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.733311 11521 status_manager.go:461] Failed to get status for pod "kube-apiserver-gernot-pi3_kube-system(6761abf6cea6552c7b386552482c777c)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.741506 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.742096 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.758507 11521 status_manager.go:461] Failed to get status for pod "kube-controller-manager-gernot-pi3_kube-system(98578eb479695566b995bf6ae33a05fe)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.767738 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.767796 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.781327 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.782799 11521 status_manager.go:461] Failed to get status for pod "kube-scheduler-gernot-pi3_kube-system(75b1b9ab0f3a37601ee6e0a6c14cc1a7)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.794546 11521 pod_container_deletor.go:77] Container "ff22c4d0d47ddcd6d4521e8e407b70f7cf560fbce364469e40cc0e32bfd02edc" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.795477 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.808002 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/6761abf6cea6552c7b386552482c777c-ca-certs") pod "kube-apiserver-gernot-pi3" (UID: "6761abf6cea6552c7b386552482c777c")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.808304 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.808829 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/75b1b9ab0f3a37601ee6e0a6c14cc1a7-kubeconfig") pod "kube-scheduler-gernot-pi3" (UID: "75b1b9ab0f3a37601ee6e0a6c14cc1a7")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.809349 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/6761abf6cea6552c7b386552482c777c-k8s-certs") pod "kube-apiserver-gernot-pi3" (UID: "6761abf6cea6552c7b386552482c777c")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.809845 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/16846a2efbc5ef643330dac361a49ad0-etcd-certs") pod "etcd-gernot-pi3" (UID: "16846a2efbc5ef643330dac361a49ad0")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.810371 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-flexvolume-dir") pod "kube-controller-manager-gernot-pi3" (UID: "98578eb479695566b995bf6ae33a05fe")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.811506 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-k8s-certs") pod "kube-controller-manager-gernot-pi3" (UID: "98578eb479695566b995bf6ae33a05fe")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.812153 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-ca-certs") pod "kube-controller-manager-gernot-pi3" (UID: "98578eb479695566b995bf6ae33a05fe")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.812918 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-kubeconfig") pod "kube-controller-manager-gernot-pi3" (UID: "98578eb479695566b995bf6ae33a05fe")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.813603 11521 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/16846a2efbc5ef643330dac361a49ad0-etcd-data") pod "etcd-gernot-pi3" (UID: "16846a2efbc5ef643330dac361a49ad0")
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.825009 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.826821 11521 pod_container_deletor.go:77] Container "43d64042b24260544bdbeef99daa6418264d0eb0cea1831d66390353fb7bad67" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.827277 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:40 gernot-pi3 kubelet[11521]: I0505 12:18:40.841028 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:40 gernot-pi3 kubelet[11521]: W0505 12:18:40.841326 11521 pod_container_deletor.go:77] Container "4f5f4d03e5130ecbd72dc7aadb61bbcc4ee2dc0b906628af718b94cb1add48d9" not found in pod's containers
May 05 12:18:40 gernot-pi3 kubelet[11521]: E0505 12:18:40.843184 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:41 gernot-pi3 kubelet[11521]: E0505 12:18:41.078875 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:41 gernot-pi3 kubelet[11521]: E0505 12:18:41.079489 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:41 gernot-pi3 kubelet[11521]: E0505 12:18:41.088751 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:42 gernot-pi3 kubelet[11521]: E0505 12:18:42.081054 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:42 gernot-pi3 kubelet[11521]: E0505 12:18:42.094326 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:42 gernot-pi3 kubelet[11521]: E0505 12:18:42.094326 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:42 gernot-pi3 kubelet[11521]: W0505 12:18:42.613622 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:42 gernot-pi3 kubelet[11521]: E0505 12:18:42.614474 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:18:43 gernot-pi3 kubelet[11521]: E0505 12:18:43.086944 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:43 gernot-pi3 kubelet[11521]: E0505 12:18:43.097537 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:43 gernot-pi3 kubelet[11521]: E0505 12:18:43.098424 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:44 gernot-pi3 kubelet[11521]: E0505 12:18:44.089145 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:44 gernot-pi3 kubelet[11521]: E0505 12:18:44.100520 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:44 gernot-pi3 kubelet[11521]: E0505 12:18:44.102022 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:44 gernot-pi3 kubelet[11521]: I0505 12:18:44.162972 11521 kuberuntime_manager.go:757] checking backoff for container "etcd" in pod "etcd-gernot-pi3_kube-system(16846a2efbc5ef643330dac361a49ad0)"
May 05 12:18:44 gernot-pi3 kubelet[11521]: I0505 12:18:44.200772 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:44 gernot-pi3 kubelet[11521]: W0505 12:18:44.221561 11521 pod_container_deletor.go:77] Container "4016a5100d10ed613e99cf40867eb9bcdb3ebf2de5a16cb8b3136f8bd75d2037" not found in pod's containers
May 05 12:18:45 gernot-pi3 kubelet[11521]: E0505 12:18:45.091647 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:45 gernot-pi3 kubelet[11521]: E0505 12:18:45.103374 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:45 gernot-pi3 kubelet[11521]: E0505 12:18:45.110457 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:45 gernot-pi3 kubelet[11521]: I0505 12:18:45.420544 11521 kuberuntime_manager.go:757] checking backoff for container "kube-apiserver" in pod "kube-apiserver-gernot-pi3_kube-system(6761abf6cea6552c7b386552482c777c)"
May 05 12:18:45 gernot-pi3 kubelet[11521]: I0505 12:18:45.582068 11521 kuberuntime_manager.go:757] checking backoff for container "kube-scheduler" in pod "kube-scheduler-gernot-pi3_kube-system(75b1b9ab0f3a37601ee6e0a6c14cc1a7)"
May 05 12:18:45 gernot-pi3 kubelet[11521]: I0505 12:18:45.628157 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:45 gernot-pi3 kubelet[11521]: I0505 12:18:45.652593 11521 kuberuntime_manager.go:757] checking backoff for container "kube-controller-manager" in pod "kube-controller-manager-gernot-pi3_kube-system(98578eb479695566b995bf6ae33a05fe)"
May 05 12:18:45 gernot-pi3 kubelet[11521]: W0505 12:18:45.727478 11521 pod_container_deletor.go:77] Container "08cd5ad797132db9f4cf24be9bba7caaf16ce9f00b638ab7774f47bd6e2e0e66" not found in pod's containers
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.094090 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.106238 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.113468 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:46 gernot-pi3 kubelet[11521]: I0505 12:18:46.396246 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:46 gernot-pi3 kubelet[11521]: W0505 12:18:46.440936 11521 pod_container_deletor.go:77] Container "486b62caad05feacd93f72550abb51336311b885702096103861b6a4319776ae" not found in pod's containers
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.503939 11521 remote_runtime.go:278] ContainerStatus "93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.504281 11521 kuberuntime_container.go:385] ContainerStatus for 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de error: rpc error: code = Unknown desc = Error: No such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.504413 11521 kuberuntime_manager.go:873] getPodContainerStatuses for pod "kube-controller-manager-gernot-pi3_kube-system(98578eb479695566b995bf6ae33a05fe)" failed: rpc error: code = Unknown desc = Error: No such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de
May 05 12:18:46 gernot-pi3 kubelet[11521]: E0505 12:18:46.504590 11521 generic.go:241] PLEG: Ignoring events for pod kube-controller-manager-gernot-pi3/kube-system: rpc error: code = Unknown desc = Error: No such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.106022 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.109389 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.116041 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:47 gernot-pi3 kubelet[11521]: I0505 12:18:47.254028 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:47 gernot-pi3 kubelet[11521]: I0505 12:18:47.342428 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.396747 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.614104 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:18:47 gernot-pi3 kubelet[11521]: W0505 12:18:47.632000 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:47 gernot-pi3 kubelet[11521]: E0505 12:18:47.633021 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:18:48 gernot-pi3 kubelet[11521]: E0505 12:18:48.109015 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:48 gernot-pi3 kubelet[11521]: E0505 12:18:48.119181 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:48 gernot-pi3 kubelet[11521]: E0505 12:18:48.119276 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:48 gernot-pi3 kubelet[11521]: I0505 12:18:48.134386 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:48 gernot-pi3 kubelet[11521]: W0505 12:18:48.181310 11521 status_manager.go:461] Failed to get status for pod "etcd-gernot-pi3_kube-system(16846a2efbc5ef643330dac361a49ad0)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/etcd-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused
May 05 12:18:48 gernot-pi3 kubelet[11521]: I0505 12:18:48.273330 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:48 gernot-pi3 kubelet[11521]: I0505 12:18:48.365791 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:49 gernot-pi3 kubelet[11521]: I0505 12:18:49.995996 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:50 gernot-pi3 kubelet[11521]: I0505 12:18:50.072496 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:50 gernot-pi3 kubelet[11521]: I0505 12:18:50.152310 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:50 gernot-pi3 kubelet[11521]: I0505 12:18:50.152323 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:51 gernot-pi3 kubelet[11521]: I0505 12:18:51.206504 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:51 gernot-pi3 kubelet[11521]: I0505 12:18:51.206503 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:51 gernot-pi3 kubelet[11521]: I0505 12:18:51.206504 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:52 gernot-pi3 kubelet[11521]: W0505 12:18:52.638276 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:52 gernot-pi3 kubelet[11521]: E0505 12:18:52.639098 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:18:54 gernot-pi3 kubelet[11521]: I0505 12:18:54.399361 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:18:54 gernot-pi3 kubelet[11521]: I0505 12:18:54.415831 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:18:57 gernot-pi3 kubelet[11521]: E0505 12:18:57.614605 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:18:57 gernot-pi3 kubelet[11521]: W0505 12:18:57.644029 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:18:57 gernot-pi3 kubelet[11521]: E0505 12:18:57.647301 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:18:58 gernot-pi3 kubelet[11521]: W0505 12:18:58.303712 11521 status_manager.go:461] Failed to get status for pod "kube-scheduler-gernot-pi3_kube-system(75b1b9ab0f3a37601ee6e0a6c14cc1a7)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-gernot-pi3: net/http: TLS handshake timeout
May 05 12:18:59 gernot-pi3 kubelet[11521]: E0505 12:18:59.114419 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:18:59 gernot-pi3 kubelet[11521]: E0505 12:18:59.122839 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:18:59 gernot-pi3 kubelet[11521]: E0505 12:18:59.126590 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:18:59 gernot-pi3 kubelet[11521]: E0505 12:18:59.958393 11521 event.go:209] Unable to write event: 'Post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: TLS handshake timeout' (may retry after sleeping)
May 05 12:19:02 gernot-pi3 kubelet[11521]: W0505 12:19:02.654760 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:02 gernot-pi3 kubelet[11521]: E0505 12:19:02.655552 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:04 gernot-pi3 kubelet[11521]: E0505 12:19:04.418423 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: net/http: TLS handshake timeout
May 05 12:19:07 gernot-pi3 kubelet[11521]: E0505 12:19:07.617408 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:19:07 gernot-pi3 kubelet[11521]: W0505 12:19:07.660360 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:07 gernot-pi3 kubelet[11521]: E0505 12:19:07.661146 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:08 gernot-pi3 kubelet[11521]: W0505 12:19:08.308956 11521 status_manager.go:461] Failed to get status for pod "kube-apiserver-gernot-pi3_kube-system(6761abf6cea6552c7b386552482c777c)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-gernot-pi3: net/http: TLS handshake timeout
May 05 12:19:10 gernot-pi3 kubelet[11521]: E0505 12:19:10.120079 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:10 gernot-pi3 kubelet[11521]: E0505 12:19:10.125544 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:10 gernot-pi3 kubelet[11521]: E0505 12:19:10.131701 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:11 gernot-pi3 kubelet[11521]: I0505 12:19:11.418875 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:19:11 gernot-pi3 kubelet[11521]: I0505 12:19:11.436106 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:19:12 gernot-pi3 kubelet[11521]: W0505 12:19:12.665644 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:12 gernot-pi3 kubelet[11521]: E0505 12:19:12.669211 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:17 gernot-pi3 kubelet[11521]: E0505 12:19:17.618058 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:19:17 gernot-pi3 kubelet[11521]: W0505 12:19:17.673637 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:17 gernot-pi3 kubelet[11521]: E0505 12:19:17.674357 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:18 gernot-pi3 kubelet[11521]: W0505 12:19:18.312358 11521 status_manager.go:461] Failed to get status for pod "kube-controller-manager-gernot-pi3_kube-system(98578eb479695566b995bf6ae33a05fe)": Get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-gernot-pi3: net/http: TLS handshake timeout
May 05 12:19:19 gernot-pi3 kubelet[11521]: E0505 12:19:19.970832 11521 event.go:209] Unable to write event: 'Post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: TLS handshake timeout' (may retry after sleeping)
May 05 12:19:21 gernot-pi3 kubelet[11521]: E0505 12:19:21.135097 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:21 gernot-pi3 kubelet[11521]: E0505 12:19:21.155008 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:21 gernot-pi3 kubelet[11521]: E0505 12:19:21.155390 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:21 gernot-pi3 kubelet[11521]: E0505 12:19:21.438729 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: net/http: TLS handshake timeout
May 05 12:19:22 gernot-pi3 kubelet[11521]: W0505 12:19:22.680261 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:22 gernot-pi3 kubelet[11521]: E0505 12:19:22.683698 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:27 gernot-pi3 kubelet[11521]: E0505 12:19:27.618769 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:19:27 gernot-pi3 kubelet[11521]: W0505 12:19:27.690641 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:27 gernot-pi3 kubelet[11521]: E0505 12:19:27.691461 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:28 gernot-pi3 kubelet[11521]: I0505 12:19:28.443838 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:19:28 gernot-pi3 kubelet[11521]: I0505 12:19:28.515553 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:19:32 gernot-pi3 kubelet[11521]: E0505 12:19:32.137922 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:32 gernot-pi3 kubelet[11521]: E0505 12:19:32.264874 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:32 gernot-pi3 kubelet[11521]: E0505 12:19:32.267865 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:32 gernot-pi3 kubelet[11521]: W0505 12:19:32.698987 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:32 gernot-pi3 kubelet[11521]: E0505 12:19:32.699802 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:37 gernot-pi3 kubelet[11521]: E0505 12:19:37.632122 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:19:37 gernot-pi3 kubelet[11521]: W0505 12:19:37.707731 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:37 gernot-pi3 kubelet[11521]: E0505 12:19:37.716560 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:38 gernot-pi3 kubelet[11521]: E0505 12:19:38.545667 11521 kubelet_node_status.go:106] Unable to register node "gernot-pi3" with API server: Post https://192.168.1.123:6443/api/v1/nodes: net/http: TLS handshake timeout
May 05 12:19:39 gernot-pi3 kubelet[11521]: E0505 12:19:39.976116 11521 event.go:209] Unable to write event: 'Post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: TLS handshake timeout' (may retry after sleeping)
May 05 12:19:42 gernot-pi3 kubelet[11521]: W0505 12:19:42.739936 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:42 gernot-pi3 kubelet[11521]: E0505 12:19:42.743609 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 05 12:19:43 gernot-pi3 kubelet[11521]: E0505 12:19:43.143670 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list v1.Pod: Get https://192.168.1.123:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:43 gernot-pi3 kubelet[11521]: E0505 12:19:43.269875 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list v1.Service: Get https://192.168.1.123:6443/api/v1/services?limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:43 gernot-pi3 kubelet[11521]: E0505 12:19:43.272665 11521 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://192.168.1.123:6443/api/v1/nodes?fieldSelector=metadata.name%3Dgernot-pi3&limit=500&resourceVersion=0: net/http: TLS handshake timeout
May 05 12:19:45 gernot-pi3 kubelet[11521]: I0505 12:19:45.552866 11521 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach
May 05 12:19:45 gernot-pi3 kubelet[11521]: I0505 12:19:45.584857 11521 kubelet_node_status.go:82] Attempting to register node gernot-pi3
May 05 12:19:47 gernot-pi3 kubelet[11521]: E0505 12:19:47.637795 11521 eviction_manager.go:246] eviction manager: failed to get get summary stats: failed to get node info: node "gernot-pi3" not found
May 05 12:19:47 gernot-pi3 kubelet[11521]: W0505 12:19:47.752544 11521 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
May 05 12:19:47 gernot-pi3 kubelet[11521]: E0505 12:19:47.754399 11521 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized