RKE network plugin containers dont start

RKE version: v1.3.1

Docker version: (docker version,docker info preferred) Client: Docker Engine - Community Version: 20.10.11 API version: 1.40 Go version: go1.16.9 Git commit: dea9396 Built: Thu Nov 18 00:37:08 2021 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 19.03.6 API version: 1.40 (minimum version 1.12) Go version: go1.12.16 Git commit: 369ce74a3c Built: Thu Feb 13 01:26:21 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.12 GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2 docker-init: Version: 0.18.0 GitCommit: fec3683

Operating system and kernel: (cat /etc/os-release, uname -r preferred) cat /etc/os-release NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

uname -r 4.15.0-88-generic

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) VmWare VM

cluster.yml file: nodes:

address: 10.x.x.1 user: svc-config role: [controlplane,worker,etcd] ssh_key_path : terraform_pk
address: 10.x.x.2 user: svc-config role: [controlplane,worker,etcd] ssh_key_path : terraform_pk
address: 10.x.x.3 user: svc-config role: [controlplane,worker,etcd] ssh_key_path : terraform_pk

services: etcd: snapshot: true creation: 6h retention: 24h

addon_job_timeout: 100 system_images: etcd: "rancher/mirrored-coreos-etcd:v3.4.15-rancher1" kubernetes: "rancher/hyperkube:v1.18.20-rancher1" alpine: "rancher/rke-tools:v0.1.75" nginxproxy: "rancher/rke-tools:v0.1.75" certdownloader: "rancher/rke-tools:v0.1.75" kubernetesservicessidecar: "rancher/rke-tools:v0.1.75" kubedns: "rancher/mirrored-k8s-dns-kube-dns:1.15.2" dnsmasq: "rancher/mirrored-k8s-dns-dnsmasq-nanny:1.15.2" kubednssidecar: "rancher/mirrored-k8s-dns-sidecar:1.15.2" kubednsautoscaler: "rancher/mirrored-cluster-proportional-autoscaler:1.7.1" flannel: "rancher/mirrored-coreos-flannel:v0.12.0" flannelcni: "rancher/flannel-cni:v0.3.0-rancher6" caliconode: "rancher/mirrored-calico-node:v3.13.4" calicocni: "rancher/mirrored-calico-cni:v3.13.4" calicocontrollers: "rancher/mirrored-calico-kube-controllers:v3.13.4" calicoctl: "rancher/mirrored-calico-ctl:v3.13.4" calicoflexvol: "rancher/mirrored-calico-pod2daemon-flexvol:v3.13.4" canalnode: "rancher/mirrored-calico-node:v3.13.4" canalcni: "rancher/mirrored-calico-cni:v3.13.4" canalflannel: "rancher/mirrored-coreos-flannel:v0.12.0" canalflexvol: "rancher/mirrored-calico-pod2daemon-flexvol:v3.13.4" weavenode: "weaveworks/weave-kube:2.6.4" weavecni: "weaveworks/weave-npc:2.6.4" acicnideploycontainer: "noiro/cnideploy:5.1.1.0.1ae238a" acihostcontainer: "noiro/aci-containers-host:5.1.1.0.1ae238a" aciopflexcontainer: "noiro/opflex:5.1.1.0.1ae238a" acimcastcontainer: "noiro/opflex:5.1.1.0.1ae238a" aciopenvswitchcontainer: "noiro/openvswitch:5.1.1.0.1ae238a" acicontrollercontainer: "noiro/aci-containers-controller:5.1.1.0.1ae238a" acigbpservercontainer: "noiro/gbp-server:5.1.1.0.1ae238a" aciopflexservercontainer: "noiro/opflex-server:5.1.1.0.1ae238a" podinfracontainer: "rancher/mirrored-pause:3.1" ingress: "rancher/nginx-ingress-controller:nginx-0.35.0-rancher2" ingressbackend: "rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1" metricsserver: "rancher/mirrored-metrics-server:v0.3.6" coredns: "rancher/mirrored-coredns-coredns:1.6.9" corednsautoscaler: "rancher/mirrored-cluster-proportional-autoscaler:1.7.1" windowspodinfracontainer: "rancher/kubelet-pause:v0.1.6" nodelocal: "rancher/mirrored-k8s-dns-node-cache:1.15.7"

network: plugin: canal

Steps to Reproduce: rke up --config config.yaml

Results: Network containers (k8s_autoscaler_coredns-autoscaler, k8s_kube-flannel_canal, k8s_calico-node_canal-, k8s_POD_coredns-autoscaler, k8s_flexvol-driver_canal, k8s_install-cni_canal, k8s_POD_canal) don't start which results in creation failure.

snippet of RKE up output log: Now checking status of node , try rancher/rke#1\"", "time=\"2021-11-29T15:12:10Z\" level=error msg=\"Host 10.53.5.62 failed to report Ready status with error: [controlplane] Error getting node : \\"10.x.x.x.\\" not found\"", "time=\"2021-11-29T15:12:10Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"", "time=\"2021-11-29T15:12:10Z\" level=info msg=\"Processing controlplane host 10.x.x.x\"", "time=\"2021-11-29T15:12:10Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.x, try rancher/rke#1\"", "time=\"2021-11-29T15:12:35Z\" level=error msg=\"Failed to upgrade hosts: 10.53.5.60 with error [[controlplane] Error getting node 10.x.x.x: \\"10.x.x.x\\" not found]\"", "time=\"2021-11-29T15:12:35Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.53.5.60: \\"10.x.x.x\\" not found]]\"

Using system_images is no longer supported, please remove it from the YAML and start using kubernetes_version. This will set all the correct images for that version, you can list supported k8s versions using rke config -l -a.

If you still encounter issues after using the correct configuration, please share full log of rke up, and outputs of kubectl get nodes -o wide and kubectl get pods -A.

I removed the system images from the cluster file but I still see same problem. Network pods don't start. Here is the output from rke up.

fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["rke", "up", "--config", "./rancher-cluster.yml", "--disable-port-check"], "delta": "0:09:04.008117", "end": "2021-11-30 11:59:47.318061", "msg": "non-zero return code", "rc": 1, "start": "2021-11-30 11:50:43.309944", "stderr": "", "stderr_lines": [], "stdout": "time=\"2021-11-30T11:50:43Z\" level=info msg=\"Running RKE version: v1.3.1\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"Initiating Kubernetes cluster\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"Building Kubernetes cluster\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:50Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:51Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:51Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:50:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:52Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:52Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:57Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:58Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:58Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:50:59Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:51:00Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:51:00Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:51:00Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:51:00Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"\ntime=\"2021-11-30T11:51:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:01Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:51:01Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:01Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Reconciling cluster state\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Pre-pulling kubernetes images\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:51:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:52:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:53:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:14Z\" level=info msg=\"Kubernetes images pulled successfully\"\ntime=\"2021-11-30T11:53:14Z\" level=info msg=\"[etcd] Building up etcd plane..\"\ntime=\"2021-11-30T11:53:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:37Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:53:38Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:38Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:38Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:38Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:53:39Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:39Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:53:39Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:53:39Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:53:44Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:14Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:15Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:15Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:20Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:21Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:21Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:21Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:21Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:54:22Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:22Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:22Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:22Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:22Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:54:24Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:54:24Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:24Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:25Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:25Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:25Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:25Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:54:26Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:26Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:26Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:26Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:29Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:29Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:30Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:30Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:30Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:49Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:49Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:54Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:55Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:55Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:55Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:57Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:54:57Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:54:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:54:58Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:54:59Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:54:59Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:54:59Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:54:59Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:55:00Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:00Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:00Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:00Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:03Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:03Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:03Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:03Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:03Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:22Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:22Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:27Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:27Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:28Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:28Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:28Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:29Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:30Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:30Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:30Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:30Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:30Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:55:31Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:55:31Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"\ntime=\"2021-11-30T11:55:31Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"\ntime=\"2021-11-30T11:55:31Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T11:55:56Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-11-30T11:56:21Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-11-30T11:56:46Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"\ntime=\"2021-11-30T11:56:46Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:56Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T11:56:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:05Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:06Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:06Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:07Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:08Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:11Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T11:57:11Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[worker] Building up Worker Plane..\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:12Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:24Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T11:57:24Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:57:26Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"\ntime=\"2021-11-30T11:58:00Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T11:58:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T11:58:00Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:58:01Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:58:01Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T11:58:01Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T11:58:05Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T11:58:05Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T11:58:06Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T11:58:07Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T11:58:07Z\" level=info msg=\"[worker] Successfully started Worker Plane..\"\ntime=\"2021-11-30T11:58:07Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T11:58:32Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"\ntime=\"2021-11-30T11:58:32Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-11-30T11:58:57Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"\ntime=\"2021-11-30T11:58:57Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-11-30T11:59:22Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"\ntime=\"2021-11-30T11:59:22Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"\ntime=\"2021-11-30T11:59:22Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"\ntime=\"2021-11-30T11:59:22Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T11:59:47Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"\ntime=\"2021-11-30T11:59:47Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\"", "stdout_lines": ["time=\"2021-11-30T11:50:43Z\" level=info msg=\"Running RKE version: v1.3.1\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"Initiating Kubernetes cluster\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"Building Kubernetes cluster\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"", "time=\"2021-11-30T11:50:43Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:44Z\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.78] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:50:50Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:51Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:51Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:50:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:52Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:52Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:50:57Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:57Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:58Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:58Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:50:59Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:51:00Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:51:00Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:51:00Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:51:00Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"", "time=\"2021-11-30T11:51:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:01Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:51:01Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:01Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Reconciling cluster state\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T11:51:02Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Pre-pulling kubernetes images\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:51:02Z\" level=info msg=\"Pulling image [rancher/hyperkube:v1.20.11-rancher1] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:51:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:52:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:53:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:14Z\" level=info msg=\"Kubernetes images pulled successfully\"", "time=\"2021-11-30T11:53:14Z\" level=info msg=\"[etcd] Building up etcd plane..\"", "time=\"2021-11-30T11:53:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:37Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:53:38Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:38Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:38Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:38Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:53:39Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:39Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:53:39Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:53:39Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:53:44Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:14Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:15Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:15Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:15Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:20Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:21Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:21Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:21Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:21Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:54:22Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:22Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:22Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:22Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:22Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:54:24Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:54:24Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:24Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:25Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:25Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:25Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:25Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:54:26Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:26Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:26Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:26Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:29Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:29Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:30Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:30Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:30Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:49Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:49Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:54Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:55Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:55Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:55Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:57Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:54:57Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:54:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:54:58Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:54:59Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:54:59Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:54:59Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:54:59Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:55:00Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:00Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:00Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:00Z\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:03Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:03Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:03Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:03Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:03Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:22Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:22Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:27Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:27Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:28Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:28Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:28Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:29Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:30Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:30Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:30Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:30Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:30Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:55:31Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:55:31Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"", "time=\"2021-11-30T11:55:31Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"", "time=\"2021-11-30T11:55:31Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T11:55:56Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-11-30T11:56:21Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-11-30T11:56:46Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"", "time=\"2021-11-30T11:56:46Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:56:47Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:48Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:56Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T11:56:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:56:57Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:56:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T11:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:05Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:06Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:06Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:07Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:08Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:11Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T11:57:11Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[worker] Building up Worker Plane..\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:12Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:13Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:24Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T11:57:24Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:25Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:57:26Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"", "time=\"2021-11-30T11:58:00Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T11:58:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T11:58:00Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:58:01Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T11:58:01Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T11:58:01Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T11:58:05Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T11:58:05Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T11:58:06Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T11:58:07Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T11:58:07Z\" level=info msg=\"[worker] Successfully started Worker Plane..\"", "time=\"2021-11-30T11:58:07Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T11:58:32Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"", "time=\"2021-11-30T11:58:32Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-11-30T11:58:57Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"", "time=\"2021-11-30T11:58:57Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-11-30T11:59:22Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"", "time=\"2021-11-30T11:59:22Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"", "time=\"2021-11-30T11:59:22Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"", "time=\"2021-11-30T11:59:22Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T11:59:47Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"", "time=\"2021-11-30T11:59:47Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\""]

I'm missing the output of the other 2 commands, please also share docker logs kubelet 2>&1 from 10.x.x.3

kubectl get pods -A --kubeconfig kubeconfig.yaml No resources found

kubectl get nodes -o wide --kubeconfig kubeconfig.yaml

No resources found

rke up \"Running RKE version: v1.3.1\"\ntime=\"2021-11-30T12:55:49Z\" level=info msg=\"Initiating Kubernetes cluster\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Building Kubernetes cluster\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:56Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:57Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:57Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:57Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:57Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:58Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:55:59Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:55:59Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:00Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:01Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:01Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Reconciling cluster state\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Pre-pulling kubernetes images\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Kubernetes images pulled successfully\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"[etcd] Building up etcd plane..\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:02Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:03Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:03Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:03Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:03Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:04Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:04Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:04Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:04Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:05Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:05Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:10Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:11Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:11Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:11Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:11Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:12Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:12Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:12Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:12Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:12Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:13Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:13Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:13Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:14Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:14Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:56:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:56:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:14Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:15Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:15Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:15Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:15Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:16Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:16Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:16Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:16Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:16Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:17Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:17Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:17Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:46Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:47Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:52Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:52Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:52Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:52Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:53Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:53Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:53Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:53Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:54Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:54Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:55Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:55Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:56:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:56:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:56Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:57Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:57Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:57Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:57Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:56:58Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:58Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:58Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:58Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:58Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:56:59Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:04Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:05Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:05Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:05Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-11-30T12:57:06Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:06Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:06Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:21Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:22Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:22Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:22Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:22Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:22Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:57:23Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:57:23Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"\ntime=\"2021-11-30T12:57:24Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"\ntime=\"2021-11-30T12:57:24Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T12:57:49Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-11-30T12:58:14Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:52Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:53Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:58:54Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T12:58:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:55Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:55Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:58:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:58:57Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T12:58:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:59Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T12:58:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:58:59Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:00Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Building up Worker Plane..\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:13Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:14Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"\ntime=\"2021-11-30T12:59:33Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:34Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:34Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:34Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:34Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:34Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T12:59:35Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"\ntime=\"2021-11-30T12:59:35Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"\ntime=\"2021-11-30T13:00:14Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"\ntime=\"2021-11-30T13:00:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-11-30T13:00:15Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T13:00:15Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-11-30T13:00:15Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-11-30T13:00:16Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-11-30T13:00:16Z\" level=error msg=\"Failed to upgrade worker components on NotReady hosts, error: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: I1130 13:00:10.615491 13676 proxier.go:826] syncProxyRules took 28.141668ms]\"\ntime=\"2021-11-30T13:00:16Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T13:00:41Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"\ntime=\"2021-11-30T13:00:41Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-11-30T13:01:06Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"\ntime=\"2021-11-30T13:01:06Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-11-30T13:01:31Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"\ntime=\"2021-11-30T13:01:31Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"\ntime=\"2021-11-30T13:01:31Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"\ntime=\"2021-11-30T13:01:31Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-11-30T13:01:56Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"\ntime=\"2021-11-30T13:01:56Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\"", "stdout_lines": ["time=\"2021-11-30T12:55:49Z\" level=info msg=\"Running RKE version: v1.3.1\"", "time=\"2021-11-30T12:55:49Z\" level=info msg=\"Initiating Kubernetes cluster\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Building Kubernetes cluster\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:55:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:51Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:55:56Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:56Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:57Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:57Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:57Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:57Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:55:58Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:55:59Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:55:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:55:59Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:55:59Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:00Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:01Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:01Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:01Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Reconciling cluster state\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-11-30T12:56:02Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Pre-pulling kubernetes images\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Kubernetes images pulled successfully\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"[etcd] Building up etcd plane..\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:02Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:03Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:03Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:03Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:03Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:04Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:04Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:04Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:04Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:05Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:05Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:05Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:10Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:11Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:11Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:11Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:11Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:12Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:12Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:12Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:12Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:12Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:13Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:13Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:13Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:14Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:14Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:56:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:56:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:14Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:15Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:15Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:15Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:15Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:16Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:16Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:16Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:16Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:16Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:17Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:17Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:17Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:46Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:47Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:52Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:52Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:52Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:52Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:53Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:53Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:53Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:53Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:54Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:54Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:55Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:55Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:56:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:56:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:56Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:57Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:57Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:57Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:57Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:56:58Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:58Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:58Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:58Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:58Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:56:59Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:56:59Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:04Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:05Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:05Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:05Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-11-30T12:57:06Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:06Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:06Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:21Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:22Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:22Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:22Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:22Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:22Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:23Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:23Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:57:23Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:57:23Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"", "time=\"2021-11-30T12:57:24Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"", "time=\"2021-11-30T12:57:24Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T12:57:49Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-11-30T12:58:14Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:39Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:40Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:49Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:50Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:51Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:52Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:53Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"", "time=\"2021-11-30T12:58:54Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T12:58:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:55Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:55Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:58:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:58:57Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T12:58:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:57Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:58Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:59Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T12:58:59Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:58:59Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:00Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Building up Worker Plane..\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:01Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:13Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:14Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"", "time=\"2021-11-30T12:59:33Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:34Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:34Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:34Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:34Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:34Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T12:59:35Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"", "time=\"2021-11-30T12:59:35Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"", "time=\"2021-11-30T13:00:14Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"", "time=\"2021-11-30T13:00:14Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-11-30T13:00:15Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T13:00:15Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-11-30T13:00:15Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-11-30T13:00:16Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-11-30T13:00:16Z\" level=error msg=\"Failed to upgrade worker components on NotReady hosts, error: [Failed to verify healthcheck: Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: I1130 13:00:10.615491 13676 proxier.go:826] syncProxyRules took 28.141668ms]\"", "time=\"2021-11-30T13:00:16Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T13:00:41Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"", "time=\"2021-11-30T13:00:41Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-11-30T13:01:06Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"", "time=\"2021-11-30T13:01:06Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-11-30T13:01:31Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"", "time=\"2021-11-30T13:01:31Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"", "time=\"2021-11-30T13:01:31Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"", "time=\"2021-11-30T13:01:31Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-11-30T13:01:56Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"", "time=\"2021-11-30T13:01:56Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\""]}

docker logs kubelet 2>&1

echo kubelet --streaming-connection-idle-timeout=30m --v=2 --address=0.0.0.0 --anonymous-auth=false --cni-bin-dir=/opt/cni/bin --cloud-provider= --root-dir=/var/lib/kubelet --read-only-port=0 --authentication-token-webhook=true --cni-conf-dir=/etc/cni/net.d --cgroups-per-qos=True --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --fail-swap-on=false --network-plugin=cni --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --resolv-conf=/etc/resolv.conf --hostname-override=10.x.x.3 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --pod-infra-container-image=rancher/mirrored-pause:3.2 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --make-iptables-util-chains=true --cluster-dns=10.43.0.10 --authorization-mode=Webhook --event-qps=0
grep -q -- --resolv-conf=/etc/resolv.conf
pgrep -f systemd-resolved
'[' -f /run/systemd/resolve/resolv.conf ']'
RESOLVCONF=--resolv-conf=/run/systemd/resolve/resolv.conf
'[' '!' -z '' ']' ++ /opt/rke-tools/bin/docker info ++ grep -i 'cgroup driver' ++ awk '{print $3}' WARNING: No swap limit support
CGROUPDRIVER=cgroupfs
'[' '' == true ']'
exec kubelet --streaming-connection-idle-timeout=30m --v=2 --address=0.0.0.0 --anonymous-auth=false --cni-bin-dir=/opt/cni/bin --cloud-provider= --root-dir=/var/lib/kubelet --read-only-port=0 --authentication-token-webhook=true --cni-conf-dir=/etc/cni/net.d --cgroups-per-qos=True --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --fail-swap-on=false --network-plugin=cni --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --resolv-conf=/etc/resolv.conf --hostname-override=10.x.x.3 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --pod-infra-container-image=rancher/mirrored-pause:3.2 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --make-iptables-util-chains=true --cluster-dns=10.43.0.10 --authorization-mode=Webhook --event-qps=0 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf Flag --streaming-connection-idle-timeout has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet. Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --cgroups-per-qos has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --tls-cipher-suites has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --make-iptables-util-chains has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --event-qps has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. I1130 12:59:03.268407 12526 flags.go:59] FLAG: --add-dir-header="false" I1130 12:59:03.268484 12526 flags.go:59] FLAG: --address="0.0.0.0" I1130 12:59:03.268490 12526 flags.go:59] FLAG: --allowed-unsafe-sysctls="[]" I1130 12:59:03.268503 12526 flags.go:59] FLAG: --alsologtostderr="false" I1130 12:59:03.268507 12526 flags.go:59] FLAG: --anonymous-auth="false" I1130 12:59:03.268511 12526 flags.go:59] FLAG: --application-metrics-count-limit="100" I1130 12:59:03.268515 12526 flags.go:59] FLAG: --authentication-token-webhook="true" I1130 12:59:03.268531 12526 flags.go:59] FLAG: --authentication-token-webhook-cache-ttl="2m0s" I1130 12:59:03.268536 12526 flags.go:59] FLAG: --authorization-mode="Webhook" I1130 12:59:03.268540 12526 flags.go:59] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" I1130 12:59:03.268544 12526 flags.go:59] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" I1130 12:59:03.268547 12526 flags.go:59] FLAG: --azure-container-registry-config="" I1130 12:59:03.268561 12526 flags.go:59] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" I1130 12:59:03.268566 12526 flags.go:59] FLAG: --bootstrap-kubeconfig="" I1130 12:59:03.268569 12526 flags.go:59] FLAG: --cert-dir="/var/lib/kubelet/pki" I1130 12:59:03.268572 12526 flags.go:59] FLAG: --cgroup-driver="cgroupfs" I1130 12:59:03.268576 12526 flags.go:59] FLAG: --cgroup-root="" I1130 12:59:03.268579 12526 flags.go:59] FLAG: --cgroups-per-qos="true" I1130 12:59:03.268582 12526 flags.go:59] FLAG: --chaos-chance="0" I1130 12:59:03.268590 12526 flags.go:59] FLAG: --client-ca-file="/etc/kubernetes/ssl/kube-ca.pem" I1130 12:59:03.268594 12526 flags.go:59] FLAG: --cloud-config="" I1130 12:59:03.268597 12526 flags.go:59] FLAG: --cloud-provider="" I1130 12:59:03.268600 12526 flags.go:59] FLAG: --cluster-dns="[10.43.0.10]" I1130 12:59:03.268611 12526 flags.go:59] FLAG: --cluster-domain="cluster.local" I1130 12:59:03.268614 12526 flags.go:59] FLAG: --cni-bin-dir="/opt/cni/bin" I1130 12:59:03.268617 12526 flags.go:59] FLAG: --cni-cache-dir="/var/lib/cni/cache" I1130 12:59:03.268623 12526 flags.go:59] FLAG: --cni-conf-dir="/etc/cni/net.d" I1130 12:59:03.268629 12526 flags.go:59] FLAG: --config="" I1130 12:59:03.268633 12526 flags.go:59] FLAG: --container-hints="/etc/cadvisor/container_hints.json" I1130 12:59:03.268637 12526 flags.go:59] FLAG: --container-log-max-files="5" I1130 12:59:03.268641 12526 flags.go:59] FLAG: --container-log-max-size="10Mi" I1130 12:59:03.268644 12526 flags.go:59] FLAG: --container-runtime="docker" I1130 12:59:03.268647 12526 flags.go:59] FLAG: --container-runtime-endpoint="unix:///var/run/dockershim.sock" I1130 12:59:03.268651 12526 flags.go:59] FLAG: --containerd="/run/containerd/containerd.sock" I1130 12:59:03.268655 12526 flags.go:59] FLAG: --containerd-namespace="k8s.io" I1130 12:59:03.268658 12526 flags.go:59] FLAG: --contention-profiling="false" I1130 12:59:03.268662 12526 flags.go:59] FLAG: --cpu-cfs-quota="true" I1130 12:59:03.268665 12526 flags.go:59] FLAG: --cpu-cfs-quota-period="100ms" I1130 12:59:03.268669 12526 flags.go:59] FLAG: --cpu-manager-policy="none" I1130 12:59:03.268672 12526 flags.go:59] FLAG: --cpu-manager-reconcile-period="10s" I1130 12:59:03.268675 12526 flags.go:59] FLAG: --docker="unix:///var/run/docker.sock" I1130 12:59:03.268679 12526 flags.go:59] FLAG: --docker-endpoint="unix:///var/run/docker.sock" I1130 12:59:03.268682 12526 flags.go:59] FLAG: --docker-env-metadata-whitelist="" I1130 12:59:03.268685 12526 flags.go:59] FLAG: --docker-only="false" I1130 12:59:03.268689 12526 flags.go:59] FLAG: --docker-root="/var/lib/docker" I1130 12:59:03.268692 12526 flags.go:59] FLAG: --docker-tls="false" I1130 12:59:03.268697 12526 flags.go:59] FLAG: --docker-tls-ca="ca.pem" I1130 12:59:03.268703 12526 flags.go:59] FLAG: --docker-tls-cert="cert.pem" I1130 12:59:03.268706 12526 flags.go:59] FLAG: --docker-tls-key="key.pem" I1130 12:59:03.268709 12526 flags.go:59] FLAG: --dynamic-config-dir="" I1130 12:59:03.268714 12526 flags.go:59] FLAG: --enable-cadvisor-json-endpoints="false" I1130 12:59:03.268717 12526 flags.go:59] FLAG: --enable-controller-attach-detach="true" I1130 12:59:03.268720 12526 flags.go:59] FLAG: --enable-debugging-handlers="true" I1130 12:59:03.268724 12526 flags.go:59] FLAG: --enable-load-reader="false" I1130 12:59:03.268727 12526 flags.go:59] FLAG: --enable-server="true" I1130 12:59:03.268730 12526 flags.go:59] FLAG: --enforce-node-allocatable="[pods]" I1130 12:59:03.268739 12526 flags.go:59] FLAG: --event-burst="10" I1130 12:59:03.268742 12526 flags.go:59] FLAG: --event-qps="0" I1130 12:59:03.268746 12526 flags.go:59] FLAG: --event-storage-age-limit="default=0" I1130 12:59:03.268749 12526 flags.go:59] FLAG: --event-storage-event-limit="default=0" I1130 12:59:03.268752 12526 flags.go:59] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%" I1130 12:59:03.268765 12526 flags.go:59] FLAG: --eviction-max-pod-grace-period="0" I1130 12:59:03.268769 12526 flags.go:59] FLAG: --eviction-minimum-reclaim="" I1130 12:59:03.268775 12526 flags.go:59] FLAG: --eviction-pressure-transition-period="5m0s" I1130 12:59:03.268781 12526 flags.go:59] FLAG: --eviction-soft="" I1130 12:59:03.268786 12526 flags.go:59] FLAG: --eviction-soft-grace-period="" I1130 12:59:03.268790 12526 flags.go:59] FLAG: --exit-on-lock-contention="false" I1130 12:59:03.268793 12526 flags.go:59] FLAG: --experimental-allocatable-ignore-eviction="false" I1130 12:59:03.268796 12526 flags.go:59] FLAG: --experimental-bootstrap-kubeconfig="" I1130 12:59:03.268799 12526 flags.go:59] FLAG: --experimental-check-node-capabilities-before-mount="false" I1130 12:59:03.268803 12526 flags.go:59] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim" I1130 12:59:03.268806 12526 flags.go:59] FLAG: --experimental-kernel-memcg-notification="false" I1130 12:59:03.268809 12526 flags.go:59] FLAG: --experimental-logging-sanitization="false" I1130 12:59:03.268812 12526 flags.go:59] FLAG: --experimental-mounter-path="" I1130 12:59:03.268816 12526 flags.go:59] FLAG: --fail-swap-on="false" I1130 12:59:03.268819 12526 flags.go:59] FLAG: --feature-gates="" I1130 12:59:03.268823 12526 flags.go:59] FLAG: --file-check-frequency="20s" I1130 12:59:03.268827 12526 flags.go:59] FLAG: --global-housekeeping-interval="1m0s" I1130 12:59:03.268830 12526 flags.go:59] FLAG: --hairpin-mode="promiscuous-bridge" I1130 12:59:03.268834 12526 flags.go:59] FLAG: --healthz-bind-address="127.0.0.1" I1130 12:59:03.268838 12526 flags.go:59] FLAG: --healthz-port="10248" I1130 12:59:03.268841 12526 flags.go:59] FLAG: --help="false" I1130 12:59:03.268844 12526 flags.go:59] FLAG: --hostname-override="10.x.x.3" I1130 12:59:03.268847 12526 flags.go:59] FLAG: --housekeeping-interval="10s" I1130 12:59:03.268853 12526 flags.go:59] FLAG: --http-check-frequency="20s" I1130 12:59:03.268856 12526 flags.go:59] FLAG: --image-credential-provider-bin-dir="" I1130 12:59:03.268859 12526 flags.go:59] FLAG: --image-credential-provider-config="" I1130 12:59:03.268865 12526 flags.go:59] FLAG: --image-gc-high-threshold="85" I1130 12:59:03.268871 12526 flags.go:59] FLAG: --image-gc-low-threshold="80" I1130 12:59:03.268877 12526 flags.go:59] FLAG: --image-pull-progress-deadline="1m0s" I1130 12:59:03.268882 12526 flags.go:59] FLAG: --image-service-endpoint="" I1130 12:59:03.268887 12526 flags.go:59] FLAG: --iptables-drop-bit="15" I1130 12:59:03.268890 12526 flags.go:59] FLAG: --iptables-masquerade-bit="14" I1130 12:59:03.268893 12526 flags.go:59] FLAG: --keep-terminated-pod-volumes="false" I1130 12:59:03.268896 12526 flags.go:59] FLAG: --kernel-memcg-notification="false" I1130 12:59:03.268900 12526 flags.go:59] FLAG: --kube-api-burst="10" I1130 12:59:03.268903 12526 flags.go:59] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" I1130 12:59:03.268907 12526 flags.go:59] FLAG: --kube-api-qps="5" I1130 12:59:03.268910 12526 flags.go:59] FLAG: --kube-reserved="" I1130 12:59:03.268913 12526 flags.go:59] FLAG: --kube-reserved-cgroup="" I1130 12:59:03.268916 12526 flags.go:59] FLAG: --kubeconfig="/etc/kubernetes/ssl/kubecfg-kube-node.yaml" I1130 12:59:03.268920 12526 flags.go:59] FLAG: --kubelet-cgroups="" I1130 12:59:03.268923 12526 flags.go:59] FLAG: --lock-file="" I1130 12:59:03.268931 12526 flags.go:59] FLAG: --log-backtrace-at=":0" I1130 12:59:03.268942 12526 flags.go:59] FLAG: --log-cadvisor-usage="false" I1130 12:59:03.268947 12526 flags.go:59] FLAG: --log-dir="" I1130 12:59:03.268952 12526 flags.go:59] FLAG: --log-file="" I1130 12:59:03.268961 12526 flags.go:59] FLAG: --log-file-max-size="1800" I1130 12:59:03.268967 12526 flags.go:59] FLAG: --log-flush-frequency="5s" I1130 12:59:03.268973 12526 flags.go:59] FLAG: --logging-format="text" I1130 12:59:03.268978 12526 flags.go:59] FLAG: --logtostderr="true" I1130 12:59:03.268983 12526 flags.go:59] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" I1130 12:59:03.268987 12526 flags.go:59] FLAG: --make-iptables-util-chains="true" I1130 12:59:03.268991 12526 flags.go:59] FLAG: --manifest-url="" I1130 12:59:03.268994 12526 flags.go:59] FLAG: --manifest-url-header="" I1130 12:59:03.269000 12526 flags.go:59] FLAG: --master-service-namespace="default" I1130 12:59:03.269003 12526 flags.go:59] FLAG: --max-open-files="1000000" I1130 12:59:03.269016 12526 flags.go:59] FLAG: --max-pods="110" I1130 12:59:03.269020 12526 flags.go:59] FLAG: --maximum-dead-containers="-1" I1130 12:59:03.269023 12526 flags.go:59] FLAG: --maximum-dead-containers-per-container="1" I1130 12:59:03.269026 12526 flags.go:59] FLAG: --minimum-container-ttl-duration="0s" I1130 12:59:03.269029 12526 flags.go:59] FLAG: --minimum-image-ttl-duration="2m0s" I1130 12:59:03.269033 12526 flags.go:59] FLAG: --network-plugin="cni" I1130 12:59:03.269038 12526 flags.go:59] FLAG: --network-plugin-mtu="0" I1130 12:59:03.269041 12526 flags.go:59] FLAG: --node-ip="" I1130 12:59:03.269044 12526 flags.go:59] FLAG: --node-labels="" I1130 12:59:03.269049 12526 flags.go:59] FLAG: --node-status-max-images="50" I1130 12:59:03.269052 12526 flags.go:59] FLAG: --node-status-update-frequency="10s" I1130 12:59:03.269055 12526 flags.go:59] FLAG: --non-masquerade-cidr="10.0.0.0/8" I1130 12:59:03.269058 12526 flags.go:59] FLAG: --one-output="false" I1130 12:59:03.269062 12526 flags.go:59] FLAG: --oom-score-adj="-999" I1130 12:59:03.269065 12526 flags.go:59] FLAG: --pod-cidr="" I1130 12:59:03.269068 12526 flags.go:59] FLAG: --pod-infra-container-image="rancher/mirrored-pause:3.2" I1130 12:59:03.269072 12526 flags.go:59] FLAG: --pod-manifest-path="" I1130 12:59:03.269075 12526 flags.go:59] FLAG: --pod-max-pids="-1" I1130 12:59:03.269079 12526 flags.go:59] FLAG: --pods-per-core="0" I1130 12:59:03.269082 12526 flags.go:59] FLAG: --port="10250" I1130 12:59:03.269085 12526 flags.go:59] FLAG: --protect-kernel-defaults="false" I1130 12:59:03.269088 12526 flags.go:59] FLAG: --provider-id="" I1130 12:59:03.269091 12526 flags.go:59] FLAG: --qos-reserved="" I1130 12:59:03.269095 12526 flags.go:59] FLAG: --read-only-port="0" I1130 12:59:03.269098 12526 flags.go:59] FLAG: --really-crash-for-testing="false" I1130 12:59:03.269107 12526 flags.go:59] FLAG: --redirect-container-streaming="false" I1130 12:59:03.269110 12526 flags.go:59] FLAG: --register-node="true" I1130 12:59:03.269113 12526 flags.go:59] FLAG: --register-schedulable="true" I1130 12:59:03.269117 12526 flags.go:59] FLAG: --register-with-taints="" I1130 12:59:03.269130 12526 flags.go:59] FLAG: --registry-burst="10" I1130 12:59:03.269134 12526 flags.go:59] FLAG: --registry-qps="5" I1130 12:59:03.269137 12526 flags.go:59] FLAG: --reserved-cpus="" I1130 12:59:03.269140 12526 flags.go:59] FLAG: --resolv-conf="/run/systemd/resolve/resolv.conf" I1130 12:59:03.269144 12526 flags.go:59] FLAG: --root-dir="/var/lib/kubelet" I1130 12:59:03.269147 12526 flags.go:59] FLAG: --rotate-certificates="false" I1130 12:59:03.269150 12526 flags.go:59] FLAG: --rotate-server-certificates="false" I1130 12:59:03.269153 12526 flags.go:59] FLAG: --runonce="false" I1130 12:59:03.269157 12526 flags.go:59] FLAG: --runtime-cgroups="" I1130 12:59:03.269160 12526 flags.go:59] FLAG: --runtime-request-timeout="2m0s" I1130 12:59:03.269163 12526 flags.go:59] FLAG: --seccomp-profile-root="/var/lib/kubelet/seccomp" I1130 12:59:03.269166 12526 flags.go:59] FLAG: --serialize-image-pulls="true" I1130 12:59:03.269170 12526 flags.go:59] FLAG: --skip-headers="false" I1130 12:59:03.269173 12526 flags.go:59] FLAG: --skip-log-headers="false" I1130 12:59:03.269176 12526 flags.go:59] FLAG: --stderrthreshold="2" I1130 12:59:03.269179 12526 flags.go:59] FLAG: --storage-driver-buffer-duration="1m0s" I1130 12:59:03.269185 12526 flags.go:59] FLAG: --storage-driver-db="cadvisor" I1130 12:59:03.269188 12526 flags.go:59] FLAG: --storage-driver-host="localhost:8086" I1130 12:59:03.269192 12526 flags.go:59] FLAG: --storage-driver-password="root" I1130 12:59:03.269195 12526 flags.go:59] FLAG: --storage-driver-secure="false" I1130 12:59:03.269201 12526 flags.go:59] FLAG: --storage-driver-table="stats" I1130 12:59:03.269204 12526 flags.go:59] FLAG: --storage-driver-user="root" I1130 12:59:03.269208 12526 flags.go:59] FLAG: --streaming-connection-idle-timeout="30m0s" I1130 12:59:03.269211 12526 flags.go:59] FLAG: --sync-frequency="1m0s" I1130 12:59:03.269214 12526 flags.go:59] FLAG: --system-cgroups="" I1130 12:59:03.269217 12526 flags.go:59] FLAG: --system-reserved="" I1130 12:59:03.269220 12526 flags.go:59] FLAG: --system-reserved-cgroup="" I1130 12:59:03.269224 12526 flags.go:59] FLAG: --tls-cert-file="" I1130 12:59:03.269227 12526 flags.go:59] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305]" I1130 12:59:03.269244 12526 flags.go:59] FLAG: --tls-min-version="" I1130 12:59:03.269249 12526 flags.go:59] FLAG: --tls-private-key-file="" I1130 12:59:03.269252 12526 flags.go:59] FLAG: --topology-manager-policy="none" I1130 12:59:03.269255 12526 flags.go:59] FLAG: --topology-manager-scope="container" I1130 12:59:03.269258 12526 flags.go:59] FLAG: --v="2" I1130 12:59:03.269261 12526 flags.go:59] FLAG: --version="false" I1130 12:59:03.269269 12526 flags.go:59] FLAG: --vmodule="" I1130 12:59:03.269273 12526 flags.go:59] FLAG: --volume-plugin-dir="/var/lib/kubelet/volumeplugins" I1130 12:59:03.269276 12526 flags.go:59] FLAG: --volume-stats-agg-period="1m0s" I1130 12:59:03.269343 12526 feature_gate.go:243] feature gates: &{map[]} I1130 12:59:03.269450 12526 feature_gate.go:243] feature gates: &{map[]} I1130 12:59:03.269678 12526 mount_linux.go:193] Detected OS without systemd I1130 12:59:03.269814 12526 server.go:416] Version: v1.20.11 I1130 12:59:03.269955 12526 feature_gate.go:243] feature gates: &{map[]} I1130 12:59:03.270161 12526 feature_gate.go:243] feature gates: &{map[]} I1130 12:59:03.298461 12526 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem" I1130 12:59:03.298744 12526 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem I1130 12:59:03.299092 12526 manager.go:165] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct" I1130 12:59:03.349634 12526 fs.go:127] Filesystem UUIDs: map[49613417-d013-4d75-9a06-60da9d432bb2:/dev/dm-0 f936c82e-7380-49dd-80bf-1721214b58eb:/dev/sda2] I1130 12:59:03.349679 12526 fs.go:128] Filesystem partitions: map[/dev:{mountpoint:/dev major:0 minor:175 fsType:tmpfs blockSize:0} /dev/mapper/ubuntu--vg-ubuntu--lv:{mountpoint:/var/lib/docker major:253 minor:0 fsType:ext4 blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:25 fsType:tmpfs blockSize:0} /etc/resolv.conf:{mountpoint:/etc/resolv.conf major:0 minor:23 fsType:tmpfs blockSize:0} /host/dev/shm:{mountpoint:/host/dev/shm major:0 minor:25 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:23 fsType:tmpfs blockSize:0} /run/lock:{mountpoint:/run/lock major:0 minor:26 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:46 fsType:tmpfs blockSize:0} /run/user/52035:{mountpoint:/run/user/52035 major:0 minor:45 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:27 fsType:tmpfs blockSize:0} overlay_0-161:{mountpoint:/ major:0 minor:161 fsType:overlay blockSize:0}] I1130 12:59:03.351924 12526 nvidia.go:61] NVIDIA setup failed: no NVIDIA devices found I1130 12:59:03.359701 12526 manager.go:213] Machine: {Timestamp:2021-11-30 12:59:03.359321095 +0000 UTC m=+0.156844672 NumCores:4 NumPhysicalCores:1 NumSockets:4 CpuFrequency:2294686 MemoryCapacity:16819859456 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID: SystemUUID:87432B42-8CBA-3E06-3EC7-5B53E1E96E05 BootID:bf826ce5-af1f-45ac-bcc3-0083dad73e7c Filesystems:[{Device:/dev/mapper/ubuntu--vg-ubuntu--lv DeviceMajor:253 DeviceMinor:0 Capacity:58057719808 Type:vfs Inodes:3604480 HasInodes:true} {Device:/host/dev/shm DeviceMajor:0 DeviceMinor:25 Capacity:8409927680 Type:vfs Inodes:2053205 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:58057719808 Type:vfs Inodes:3604480 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:27 Capacity:8409927680 Type:vfs Inodes:2053205 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:25 Capacity:8409927680 Type:vfs Inodes:2053205 HasInodes:true} {Device:/run/lock DeviceMajor:0 DeviceMinor:26 Capacity:1681989632 Type:vfs Inodes:2053205 HasInodes:true} {Device:/run/user/1000 DeviceMajor:0 DeviceMinor:46 Capacity:1681989632 Type:vfs Inodes:2053205 HasInodes:true} {Device:/run/user/52035 DeviceMajor:0 DeviceMinor:45 Capacity:1681989632 Type:vfs Inodes:2053205 HasInodes:true} {Device:/dev DeviceMajor:0 DeviceMinor:175 Capacity:8377749504 Type:vfs Inodes:2045349 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:23 Capacity:1681989632 Type:vfs Inodes:2053205 HasInodes:true} {Device:/etc/resolv.conf DeviceMajor:0 DeviceMinor:23 Capacity:1681989632 Type:vfs Inodes:2053205 HasInodes:true} {Device:overlay DeviceMajor:0 DeviceMinor:161 Capacity:58057719808 Type:vfs Inodes:3604480 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:59051606016 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:107374182400 Scheduler:cfq}] NetworkDevices:[{Name:ens160 MacAddress:00:50:56:ab:23:d9 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:16819859456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:47185920 Type:Unified Level:3}] SocketID:0} {Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:47185920 Type:Unified Level:3}] SocketID:2} {Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:47185920 Type:Unified Level:3}] SocketID:4} {Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2} {Size:47185920 Type:Unified Level:3}] SocketID:6}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} I1130 12:59:03.359969 12526 manager_no_libpfm.go:28] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1130 12:59:03.377688 12526 manager.go:229] Version: {KernelVersion:4.15.0-88-generic ContainerOsVersion:Ubuntu 18.04.4 LTS DockerVersion:19.03.6 DockerAPIVersion:1.40 CadvisorVersion: CadvisorRevision:} I1130 12:59:03.377827 12526 server.go:645] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to / I1130 12:59:03.378200 12526 container_manager_linux.go:274] container manager verified user specified cgroup-root exists: [] I1130 12:59:03.378213 12526 container_manager_linux.go:279] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.1} GracePeriod:0s MinReclaim:} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.15} GracePeriod:0s MinReclaim:}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none} I1130 12:59:03.378283 12526 topology_manager.go:120] [topologymanager] Creating topology manager with none policy per container scope I1130 12:59:03.378293 12526 container_manager_linux.go:310] [topologymanager] Initializing Topology Manager with none policy and container-level scope I1130 12:59:03.378298 12526 container_manager_linux.go:315] Creating device plugin manager: true I1130 12:59:03.378305 12526 manager.go:133] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock W1130 12:59:03.378380 12526 kubelet.go:297] Using dockershim is deprecated, please consider using a full-fledged CRI implementation I1130 12:59:03.378402 12526 client.go:77] Connecting to docker on unix:///var/run/docker.sock I1130 12:59:03.378413 12526 client.go:94] Start docker client with request timeout=2m0s W1130 12:59:03.386754 12526 docker_service.go:564] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth" I1130 12:59:03.386778 12526 docker_service.go:241] Hairpin mode set to "hairpin-veth" W1130 12:59:03.386911 12526 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d W1130 12:59:03.419838 12526 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d I1130 12:59:03.419873 12526 plugins.go:168] Loaded network plugin "cni" I1130 12:59:03.419958 12526 docker_service.go:256] Docker cri networking managed by cni W1130 12:59:03.420079 12526 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d I1130 12:59:03.429623 12526 docker_service.go:263] Docker Info: &{ID:JN3T:OYKH:B4FY:UZYL:A3CE:HY2N:MZSI:D2PB:GMWM:IGG3:CGXI:4TRM Containers:7 ContainersRunning:6 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:false KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:56 OomKillDisable:true NGoroutines:61 SystemTime:2021-11-30T12:59:03.421274935Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.15.0-88-generic OperatingSystem:Ubuntu 18.04.4 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0009a81c0 NCPU:4 MemTotal:16819859456 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ausdvshazzrnc03 Labels:[] ExperimentalBuild:false ServerVersion:19.03.6 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster: Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:7b11cfaabd73bb80907dd23182b9347b4245eb5d Expected:7b11cfaabd73bb80907dd23182b9347b4245eb5d} RuncCommit:{ID:v1.0.2-0-g52b36a2 Expected:v1.0.2-0-g52b36a2} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:[WARNING: No swap limit support]} I1130 12:59:03.429767 12526 docker_service.go:276] Setting cgroupDriver to cgroupfs I1130 12:59:03.429874 12526 kubelet_dockershim.go:64] Starting the GRPC server for the docker CRI shim. I1130 12:59:03.429930 12526 docker_server.go:61] Start dockershim grpc server I1130 12:59:03.439933 12526 remote_runtime.go:62] parsed scheme: "" I1130 12:59:03.439951 12526 remote_runtime.go:62] scheme "" not registered, fallback to default scheme I1130 12:59:03.439982 12526 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 }] } I1130 12:59:03.439992 12526 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:03.440071 12526 remote_image.go:50] parsed scheme: "" I1130 12:59:03.440078 12526 remote_image.go:50] scheme "" not registered, fallback to default scheme I1130 12:59:03.440088 12526 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 }] } I1130 12:59:03.440093 12526 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:03.440119 12526 server.go:1117] Using root directory: /var/lib/kubelet I1130 12:59:03.440158 12526 kubelet.go:394] Attempting to sync node with API server I1130 12:59:03.440169 12526 kubelet.go:273] Adding apiserver pod source I1130 12:59:03.440183 12526 apiserver.go:43] Waiting for node sync before watching apiserver pods I1130 12:59:03.440370 12526 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick I1130 12:59:03.440561 12526 reflector.go:219] Starting reflector v1.Service (0s) from k8s.io/client-go/informers/factory.go:134 I1130 12:59:03.440865 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0008dc180, {CONNECTING } I1130 12:59:03.440995 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0008dc320, {CONNECTING } I1130 12:59:03.441177 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0008dc180, {READY } I1130 12:59:03.441209 12526 reflector.go:219] Starting reflector v1.Node (0s) from k8s.io/client-go/informers/factory.go:134 I1130 12:59:03.441404 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0008dc320, {READY } I1130 12:59:03.455286 12526 kuberuntime_manager.go:216] Container runtime docker initialized, version: 19.03.6, apiVersion: 1.40.0 E1130 12:59:03.462581 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Node: failed to list v1.Node: nodes "10.x.x.3" is forbidden: User "system:node" cannot list resource "nodes" in API group "" at the cluster scope E1130 12:59:03.462600 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Service: failed to list v1.Service: services is forbidden: User "system:node" cannot list resource "services" in API group "" at the cluster scope E1130 12:59:04.668763 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Service: failed to list v1.Service: services is forbidden: User "system:node" cannot list resource "services" in API group "" at the cluster scope E1130 12:59:04.871296 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Node: failed to list v1.Node: nodes "10.x.x.3" is forbidden: User "system:node" cannot list resource "nodes" in API group "" at the cluster scope E1130 12:59:06.661227 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Node: failed to list v1.Node: nodes "10.x.x.3" is forbidden: User "system:node" cannot list resource "nodes" in API group "" at the cluster scope E1130 12:59:07.347262 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Service: failed to list v1.Service: services is forbidden: User "system:node" cannot list resource "services" in API group "" at the cluster scope W1130 12:59:08.420257 12526 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d E1130 12:59:09.724621 12526 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors I1130 12:59:09.725068 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/aws-ebs" I1130 12:59:09.725088 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/gce-pd" I1130 12:59:09.725102 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/cinder" I1130 12:59:09.725109 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/azure-disk" I1130 12:59:09.725116 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/azure-file" I1130 12:59:09.725123 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/vsphere-volume" I1130 12:59:09.725133 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/empty-dir" I1130 12:59:09.725142 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/git-repo" I1130 12:59:09.725152 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/host-path" I1130 12:59:09.725160 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/nfs" I1130 12:59:09.725168 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/secret" I1130 12:59:09.725175 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/iscsi" I1130 12:59:09.725186 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/glusterfs" I1130 12:59:09.725197 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/rbd" I1130 12:59:09.725205 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/quobyte" I1130 12:59:09.725211 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/cephfs" I1130 12:59:09.725218 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/downward-api" I1130 12:59:09.725224 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/fc" I1130 12:59:09.725231 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/flocker" I1130 12:59:09.725245 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/configmap" I1130 12:59:09.725262 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/projected" I1130 12:59:09.725292 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/portworx-volume" I1130 12:59:09.725326 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/scaleio" I1130 12:59:09.725361 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/local-volume" I1130 12:59:09.725371 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/storageos" I1130 12:59:09.725424 12526 plugins.go:638] Loaded volume plugin "kubernetes.io/csi" I1130 12:59:09.725661 12526 server.go:1176] Started kubelet E1130 12:59:09.725869 12526 kubelet.go:1291] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache I1130 12:59:09.726824 12526 server.go:148] Starting to listen on 0.0.0.0:10250 I1130 12:59:09.727358 12526 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer I1130 12:59:09.727695 12526 server.go:414] Adding debug handlers to kubelet server. I1130 12:59:09.729809 12526 volume_manager.go:269] The desired_state_of_world populator starts I1130 12:59:09.729825 12526 volume_manager.go:271] Starting Kubelet Volume Manager I1130 12:59:09.731300 12526 desired_state_of_world_populator.go:142] Desired state populator starts to run I1130 12:59:09.732610 12526 reflector.go:219] Starting reflector v1.CSIDriver (0s) from k8s.io/client-go/informers/factory.go:134 E1130 12:59:09.748172 12526 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized I1130 12:59:09.748437 12526 client.go:86] parsed scheme: "unix" I1130 12:59:09.748448 12526 client.go:86] scheme "unix" not registered, fallback to default scheme I1130 12:59:09.748515 12526 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 }] } I1130 12:59:09.748526 12526 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:09.748995 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000e2d8d0, {CONNECTING } I1130 12:59:09.749068 12526 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc000e2d8d0, {READY } I1130 12:59:09.749721 12526 factory.go:137] Registering containerd factory I1130 12:59:09.749826 12526 factory.go:55] Registering systemd factory E1130 12:59:09.750248 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc54634108af6f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e76b3fcd6f, ext:6523125131, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e76b3fcd6f, ext:6523125131, loc:(time.Location)(0x70f21a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:node" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!) I1130 12:59:09.750915 12526 csi_plugin.go:1039] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "10.x.x.3" is forbidden: User "system:node" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope E1130 12:59:09.751038 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.CSIDriver: failed to list v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:node" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope E1130 12:59:09.754860 12526 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: leases.coordination.k8s.io "10.x.x.3" is forbidden: User "system:node" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease" I1130 12:59:09.763980 12526 kubelet_network_linux.go:56] Initialized IPv4 iptables rules. I1130 12:59:09.764264 12526 status_manager.go:158] Starting to sync pod status with apiserver I1130 12:59:09.764407 12526 kubelet.go:1828] Starting kubelet main sync loop. E1130 12:59:09.765292 12526 kubelet.go:1852] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful] I1130 12:59:09.764517 12526 reflector.go:219] Starting reflector v1.RuntimeClass (0s) from k8s.io/client-go/informers/factory.go:134 I1130 12:59:09.768081 12526 factory.go:370] Registering Docker factory I1130 12:59:09.768370 12526 factory.go:101] Registering Raw factory I1130 12:59:09.768581 12526 manager.go:1203] Started watching for new ooms in manager I1130 12:59:09.769224 12526 manager.go:301] Starting recovery of all containers E1130 12:59:09.778957 12526 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.RuntimeClass: failed to list v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:node" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope I1130 12:59:09.805405 12526 manager.go:306] Recovery completed E1130 12:59:09.830713 12526 kubelet.go:2263] node "10.x.x.3" not found I1130 12:59:09.830724 12526 kubelet_node_status.go:362] Setting node annotation to enable volume controller attach/detach E1130 12:59:09.865792 12526 kubelet.go:1852] skipping pod synchronization - container runtime status check may not have completed yet I1130 12:59:09.866399 12526 kubelet_node_status.go:362] Setting node annotation to enable volume controller attach/detach I1130 12:59:09.868415 12526 kubelet_node_status.go:554] Recording NodeHasSufficientMemory event message for node 10.x.x.3 I1130 12:59:09.868441 12526 kubelet_node_status.go:554] Recording NodeHasNoDiskPressure event message for node 10.x.x.3 I1130 12:59:09.868450 12526 kubelet_node_status.go:554] Recording NodeHasSufficientPID event message for node 10.x.x.3 I1130 12:59:09.868475 12526 kubelet_node_status.go:71] Attempting to register node 10.x.x.3 E1130 12:59:09.870133 12526 kubelet_node_status.go:93] Unable to register node "10.x.x.3" with API server: nodes is forbidden: User "system:node" cannot create resource "nodes" in API group "" at the cluster scope E1130 12:59:09.870304 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c0d30", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node 10.x.x.3 status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c32b30, ext:6665952076, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c32b30, ext:6665952076, loc:(time.Location)(0x70f21a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:node" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:09.871288 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c4e62", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node 10.x.x.3 status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c36c62, ext:6665968766, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c36c62, ext:6665968766, loc:(time.Location)(0x70f21a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:node" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:09.872153 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c6df3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node 10.x.x.3 status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c38bf3, ext:6665976847, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c38bf3, ext:6665976847, loc:(time.Location)(0x70f21a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:node" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!) I1130 12:59:09.895727 12526 kubelet_node_status.go:554] Recording NodeHasSufficientMemory event message for node 10.x.x.3 I1130 12:59:09.895781 12526 kubelet_node_status.go:554] Recording NodeHasNoDiskPressure event message for node 10.x.x.3 I1130 12:59:09.895793 12526 kubelet_node_status.go:554] Recording NodeHasSufficientPID event message for node 10.x.x.3 E1130 12:59:09.897742 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c6df3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node 10.x.x.3 status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c38bf3, ext:6665976847, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e77564c806, ext:6693320747, loc:(time.Location)(0x70f21a0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c6df3" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:09.898831 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c0d30", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node 10.x.x.3 status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c32b30, ext:6665952076, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e7756466ac, ext:6693295827, loc:(time.Location)(0x70f21a0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c0d30" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) I1130 12:59:09.899752 12526 cpu_manager.go:193] [cpumanager] starting with none policy I1130 12:59:09.899761 12526 cpu_manager.go:194] [cpumanager] reconciling every 10s I1130 12:59:09.899827 12526 state_mem.go:36] [cpumanager] initializing new in-memory state store E1130 12:59:09.899829 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c4e62", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node 10.x.x.3 status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c36c62, ext:6665968766, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e775649c16, ext:6693309491, loc:(time.Location)(0x70f21a0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c4e62" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) I1130 12:59:09.900061 12526 state_mem.go:88] [cpumanager] updated default cpuset: "" I1130 12:59:09.900077 12526 state_mem.go:96] [cpumanager] updated cpuset assignments: "map[]" I1130 12:59:09.900088 12526 state_checkpoint.go:136] [cpumanager] state checkpoint: restored state from checkpoint I1130 12:59:09.900143 12526 state_checkpoint.go:137] [cpumanager] state checkpoint: defaultCPUSet: I1130 12:59:09.900158 12526 policy_none.go:43] [cpumanager] none policy: Start I1130 12:59:09.901190 12526 manager.go:236] Starting Device Plugin manager W1130 12:59:09.901218 12526 manager.go:594] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found I1130 12:59:09.901539 12526 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service I1130 12:59:09.901828 12526 manager.go:278] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock" I1130 12:59:09.901968 12526 plugin_watcher.go:52] Plugin Watcher Start at /var/lib/kubelet/plugins_registry E1130 12:59:09.901995 12526 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "10.x.x.3" not found I1130 12:59:09.902111 12526 plugin_manager.go:112] The desired_state_of_world populator (plugin watcher) starts I1130 12:59:09.902131 12526 plugin_manager.go:114] Starting Kubelet Plugin Manager E1130 12:59:09.902952 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc54634b8cee2f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeAllocatableEnforced", Message:"Updated Node Allocatable limit across pods", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e775c40c2f, ext:6699564106, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e775c40c2f, ext:6699564106, loc:(time.Location)(0x70f21a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:node" cannot create resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:09.930869 12526 kubelet.go:2263] node "10.x.x.3" not found E1130 12:59:09.956572 12526 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: leases.coordination.k8s.io "10.x.x.3" is forbidden: User "system:node" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease" E1130 12:59:10.031037 12526 kubelet.go:2263] node "10.x.x.3" not found I1130 12:59:10.070307 12526 kubelet_node_status.go:362] Setting node annotation to enable volume controller attach/detach I1130 12:59:10.095020 12526 kubelet_node_status.go:554] Recording NodeHasSufficientMemory event message for node 10.x.x.3 I1130 12:59:10.095049 12526 kubelet_node_status.go:554] Recording NodeHasNoDiskPressure event message for node 10.x.x.3 I1130 12:59:10.095057 12526 kubelet_node_status.go:554] Recording NodeHasSufficientPID event message for node 10.x.x.3 I1130 12:59:10.095091 12526 kubelet_node_status.go:71] Attempting to register node 10.x.x.3 E1130 12:59:10.097446 12526 kubelet_node_status.go:93] Unable to register node "10.x.x.3" with API server: nodes is forbidden: User "system:node" cannot create resource "nodes" in API group "" at the cluster scope E1130 12:59:10.097476 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c0d30", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node 10.x.x.3 status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c32b30, ext:6665952076, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e785aa39a4, ext:6892565441, loc:(time.Location)(0x70f21a0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c0d30" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:10.098483 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c6df3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node 10.x.x.3 status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c38bf3, ext:6665976847, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e785aa818a, ext:6892583846, loc:(time.Locatio 1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c6df3" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:10.099447 12526 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"10.x.x.3.16bc5463498c4e62", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"10.x.x.3", UID:"10.x.x.3", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasNoDiskPressure", Message:"Node 10.x.x.3 status is now: NodeHasNoDiskPressure", Source:v1.EventSource{Component:"kubelet", Host:"10.x.x.3"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e773c36c62, ext:6665968766, loc:(time.Location)(0x70f21a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc061a5e785aa65ce, ext:6892576746, loc:(time.Location)(0x70f21a0)}}, Count:3, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Series:(v1.EventSeries)(nil), Action:"", Related:(v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "10.x.x.3.16bc5463498c4e62" is forbidden: User "system:node" cannot patch resource "events" in API group "" in the namespace "default"' (will not retry!) E1130 12:59:10.131228 12526 kubelet.go:2263] node "10.x.x.3" not found E1130 12:59:10.231438 12526 kubelet.go:2263] node "10.x.x.3" not found E1130 12:59:10.359809 12526 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: leases.coordination.k8s.io "10.53.5.62" is forbidden: User "system:node" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease" E1130 12:59:10.431767 12526 kubelet.go:2263] node "10.53.5.62" not found I1130 12:59:10.497626 12526 kubelet_node_status.go:362] Setting node annotation to enable volume controller attach/detach I1130 12:59:10.527945 12526 kubelet_node_status.go:554] Recording NodeHasSufficientMemory event message for node 10.53.5.62 I1130 12:59:10.527980 12526 kubelet_node_status.go:554] Recording NodeHasNoDiskPressure event message for node 10.53.5.62 I1130 12:59:10.527988 12526 kubelet_node_status.go:554] Recording NodeHasSufficientPID event message for node 10.53.5.62 I1130 12:59:10.528021 12526 kubelet_node_status.go:71] Attempting to register node 10.53.5.62 E1130 12:59:10.530932 12526 kubelet_node_status.go:93] Unable to register node "10.53.5.62" with API server: nodes is forbidden: User "system:node" cannot create resource "nodes" in API group "" at the cluster scope

Ok, this at least explains why it is happening but we also need docker logs kube-apiserver 2>&1 from the node to see if it logs why it is happening.

Is this a new cluster or was it an existing one? If it is already a new cluster does it reproduce if you start from scratch as well?

This is a new cluster and it has been failing since the start. Here are the logs

docker logs kube-apiserver 2>&1

grep -q cloud-provider=azure
echo kube-apiserver --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --anonymous-auth=false --secure-port=6443 --service-account-issuer=rke --feature-gates=ServiceAccountIssuerDiscovery=false --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-node-port-range=30000-32767 --audit-log-format=json --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --bind-address=0.0.0.0 --allow-privileged=true --audit-log-path=/var/log/kube-audit/audit-log.json --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=authorization.k8s.io/v1beta1=true --storage-backend=etcd3 --advertise-address=10.x.x.3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --insecure-port=0 --service-account-lookup=true --profiling=false --service-cluster-ip-range=10.43.0.0/16 --requestheader-username-headers=X-Remote-User --api-audiences=unknown --audit-log-maxsize=100 --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize --audit-log-maxage=30 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cloud-provider= --etcd-servers=https://10.x.x.3:2379,https://10.x.x.1:2379,https://10.x.x.2:2379 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem
'[' kube-apiserver = kubelet ']'
exec kube-apiserver --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --anonymous-auth=false --secure-port=6443 --service-account-issuer=rke --feature-gates=ServiceAccountIssuerDiscovery=false --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-node-port-range=30000-32767 --audit-log-format=json --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --bind-address=0.0.0.0 --allow-privileged=true --audit-log-path=/var/log/kube-audit/audit-log.json --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=authorization.k8s.io/v1beta1=true --storage-backend=etcd3 --advertise-address=10.x.x.3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --insecure-port=0 --service-account-lookup=true --profiling=false --service-cluster-ip-range=10.43.0.0/16 --requestheader-username-headers=X-Remote-User --api-audiences=unknown --audit-log-maxsize=100 --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize --audit-log-maxage=30 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cloud-provider= --etcd-servers=https://10.x.x.3:2379,https://10.x.x.1:2379,https://10.x.x.2:2379 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24. I1130 12:58:40.642496 1 server.go:632] external host was not specified, using 10.x.x.3 I1130 12:58:40.643066 1 server.go:182] Version: v1.20.11 I1130 12:58:41.137481 1 shared_informer.go:240] Waiting for caches to sync for node_authorizer I1130 12:58:41.139502 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I1130 12:58:41.139579 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I1130 12:58:41.142142 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I1130 12:58:41.142242 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I1130 12:58:41.143928 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.144071 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.156367 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.156518 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.166529 1 client.go:360] parsed scheme: "passthrough" I1130 12:58:41.166681 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.3:2379 0 }] } I1130 12:58:41.166744 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:58:41.167367 1 client.go:360] parsed scheme: "passthrough" I1130 12:58:41.167452 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.1:2379 0 }] } I1130 12:58:41.167549 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:58:41.168041 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.168082 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.168308 1 client.go:360] parsed scheme: "passthrough" I1130 12:58:41.168351 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.2:2379 0 }] } I1130 12:58:41.168362 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:58:41.219142 1 instance.go:289] Using reconciler: lease I1130 12:58:41.219586 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.219620 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.228317 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.228342 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.238715 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.238743 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.257142 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.257166 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.266651 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.266676 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.276044 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.276078 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.284541 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.284574 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.292366 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.292394 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.299753 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.299785 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.307679 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.307704 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.315147 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.315231 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.322934 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.323035 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.330986 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.331010 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.338366 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.338415 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.345575 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.345671 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.355976 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.356013 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.365469 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.365491 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.373908 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.373954 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.385837 1 rest.go:131] the default service ipfamily for this cluster is: IPv4 I1130 12:58:41.481833 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.481876 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.490659 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.490928 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.498823 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.498850 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.508717 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.508743 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.518455 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.518484 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.526755 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.526780 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.535243 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.535269 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.557565 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.557595 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.568025 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.568220 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.576352 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.576380 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.584829 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.584853 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.597550 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.597575 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.611081 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.611115 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.619625 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.619659 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.629553 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.629582 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.637129 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.637162 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.647699 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.647726 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.656653 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.656680 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.667798 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.667820 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.676182 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.676213 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.684230 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.684273 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.693512 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.693655 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.703258 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.703282 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.710887 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.710932 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.728487 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.729950 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.743979 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.744294 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.755660 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.755865 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.767578 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.767603 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.780376 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.780601 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.791905 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.791954 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.801982 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.802007 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.810740 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.810789 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.818995 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.819026 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.829171 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.829216 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.836749 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.836781 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.844327 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.844445 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.853560 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.853717 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.862659 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.862688 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.870125 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.870486 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.878498 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.878528 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.885802 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.885963 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.894169 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.894201 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.903469 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.903493 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.913444 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.913695 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.920897 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.921049 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.929019 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.929131 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.936927 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.936961 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.946908 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.946941 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.954423 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.954552 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.962153 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.962204 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:41.972129 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:41.972316 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] W1130 12:58:42.124616 1 genericapiserver.go:425] Skipping API batch/v2alpha1 because it has no resources. W1130 12:58:42.137323 1 genericapiserver.go:425] Skipping API discovery.k8s.io/v1alpha1 because it has no resources. I1130 12:58:42.137505 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:42.137532 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] W1130 12:58:42.152142 1 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources. W1130 12:58:42.167846 1 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W1130 12:58:42.173079 1 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W1130 12:58:42.180040 1 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W1130 12:58:42.183264 1 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. W1130 12:58:42.189365 1 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources. W1130 12:58:42.189384 1 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources. I1130 12:58:42.201002 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I1130 12:58:42.201018 1 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I1130 12:58:42.202239 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:42.202271 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:42.210171 1 client.go:360] parsed scheme: "endpoint" I1130 12:58:42.210321 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://10.x.x.3:2379 0 } {https://10.x.x.1:2379 0 } {https://10.x.x.2:2379 0 }] I1130 12:58:44.684695 1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem I1130 12:58:44.684708 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem I1130 12:58:44.685183 1 dynamic_serving_content.go:130] Starting serving-cert::/etc/kubernetes/ssl/kube-apiserver.pem::/etc/kubernetes/ssl/kube-apiserver-key.pem I1130 12:58:44.685787 1 secure_serving.go:202] Serving securely on [::]:6443 I1130 12:58:44.685855 1 tlsconfig.go:240] Starting DynamicServingCertificateController I1130 12:58:44.685965 1 controller.go:83] Starting OpenAPI AggregationController I1130 12:58:44.686073 1 customresource_discovery_controller.go:209] Starting DiscoveryController I1130 12:58:44.686157 1 available_controller.go:475] Starting AvailableConditionController I1130 12:58:44.686221 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I1130 12:58:44.686300 1 autoregister_controller.go:141] Starting autoregister controller I1130 12:58:44.686347 1 cache.go:32] Waiting for caches to sync for autoregister controller I1130 12:58:44.686409 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/ssl/kube-ca.pem I1130 12:58:44.686434 1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem I1130 12:58:44.686507 1 naming_controller.go:291] Starting NamingConditionController I1130 12:58:44.686583 1 establishing_controller.go:76] Starting EstablishingController I1130 12:58:44.686655 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController I1130 12:58:44.686722 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I1130 12:58:44.686787 1 crd_finalizer.go:266] Starting CRDFinalizer I1130 12:58:44.686871 1 crdregistration_controller.go:111] Starting crd-autoregister controller I1130 12:58:44.686957 1 shared_informer.go:240] Waiting for caches to sync for crd-autoregister I1130 12:58:44.686391 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I1130 12:58:44.687102 1 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller I1130 12:58:44.686509 1 controller.go:86] Starting OpenAPI controller I1130 12:58:44.686527 1 apiservice_controller.go:97] Starting APIServiceRegistrationController I1130 12:58:44.687358 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I1130 12:58:44.687056 1 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem::/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem I1130 12:58:44.688173 1 apf_controller.go:261] Starting API Priority and Fairness config controller E1130 12:58:44.713990 1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/10.x.x.3, ResourceVersion: 0, AdditionalErrorMsg: I1130 12:58:44.738434 1 shared_informer.go:247] Caches are synced for node_authorizer I1130 12:58:44.786307 1 cache.go:39] Caches are synced for AvailableConditionController controller I1130 12:58:44.786433 1 cache.go:39] Caches are synced for autoregister controller I1130 12:58:44.787788 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I1130 12:58:44.788639 1 shared_informer.go:247] Caches are synced for crd-autoregister I1130 12:58:44.789153 1 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller I1130 12:58:44.790488 1 apf_controller.go:266] Running API Priority and Fairness config worker I1130 12:58:45.684662 1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I1130 12:58:45.684702 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I1130 12:58:45.693316 1 storage_scheduling.go:148] all system priority classes are created successfully or already exist. I1130 12:58:46.463098 1 controller.go:609] quota admission added evaluator for: roles.rbac.authorization.k8s.io I1130 12:58:46.540623 1 controller.go:609] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io W1130 12:58:46.730580 1 lease.go:233] Resetting endpoints for master service "kubernetes" to [10.x.x.1 10.x.x.2 10.x.x.3] I1130 12:58:46.732004 1 controller.go:609] quota admission added evaluator for: endpoints I1130 12:58:46.737688 1 controller.go:609] quota admission added evaluator for: endpointslices.discovery.k8s.io I1130 12:58:55.015465 1 controller.go:609] quota admission added evaluator for: leases.coordination.k8s.io I1130 12:59:15.623589 1 client.go:360] parsed scheme: "passthrough" I1130 12:59:15.623729 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.2:2379 0 }] } I1130 12:59:15.623858 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:19.618382 1 client.go:360] parsed scheme: "passthrough" I1130 12:59:19.618433 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.3:2379 0 }] } I1130 12:59:19.618443 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:23.716592 1 trace.go:205] Trace[82000547]: "Get" url:/api/v1/namespaces/default,user-agent:kube-apiserver/v1.20.11 (linux/amd64) kubernetes/27522a2,client:::1 (30-Nov-2021 12:59:16.719) (total time: 6997ms): Trace[82000547]: ---"About to write a response" 6997ms (12:59:00.716) Trace[82000547]: [6.997262496s] [6.997262496s] END I1130 12:59:23.717822 1 trace.go:205] Trace[1485666288]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.20.11 (linux/amd64) kubernetes/27522a2/leader-election,client:127.0.0.1 (30-Nov-2021 12:59:15.167) (total time: 8549ms): Trace[1485666288]: ---"About to write a response" 8549ms (12:59:00.717) Trace[1485666288]: [8.549900939s] [8.549900939s] END I1130 12:59:25.063762 1 client.go:360] parsed scheme: "passthrough" I1130 12:59:25.063808 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.1:2379 0 }] } I1130 12:59:25.063817 1 clientconn.go:948] ClientConn switching balancer to "pick_first" E1130 12:59:26.366045 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"} I1130 12:59:26.366333 1 trace.go:205] Trace[635405073]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.20.11 (linux/amd64) kubernetes/27522a2/leader-election,client:127.0.0.1 (30-Nov-2021 12:59:16.369) (total time: 9997ms): Trace[635405073]: [9.997282944s] [9.997282944s] END E1130 12:59:33.939808 1 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: leader changed"} I1130 12:59:33.940194 1 trace.go:205] Trace[1043114195]: "Get" url:/api/v1/namespaces/default/services/kubernetes,user-agent:kube-apiserver/v1.20.11 (linux/amd64) kubernetes/27522a2,client:::1 (30-Nov-2021 12:59:23.719) (total time: 10220ms): Trace[1043114195]: [10.220474056s] [10.220474056s] END I1130 12:59:33.961756 1 trace.go:205] Trace[1651535836]: "GuaranteedUpdate etcd3" type:*coordination.Lease (30-Nov-2021 12:59:25.746) (total time: 8214ms): Trace[1651535836]: ---"Transaction committed" 8214ms (12:59:00.961) Trace[1651535836]: [8.214773862s] [8.214773862s] END I1130 12:59:33.962035 1 trace.go:205] Trace[657834222]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.20.11 (linux/amd64) kubernetes/27522a2/leader-election,client:127.0.0.1 (30-Nov-2021 12:59:25.746) (total time: 8215ms): Trace[657834222]: ---"Object stored in database" 8214ms (12:59:00.961) Trace[657834222]: [8.215216513s] [8.215216513s] END E1130 12:59:33.964175 1 controller.go:223] unable to sync kubernetes service: Service "kubernetes" is invalid: spec.clusterIPs: Invalid value: []string{"10.43.0.1"}: failed to allocated ip:10.43.0.1 with error:provided IP is already allocated E1130 12:59:38.714540 1 status.go:71] apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: request timed out"} I1130 12:59:38.714802 1 trace.go:205] Trace[1517048506]: "Get" url:/api/v1/nodes/10.x.x.3,user-agent:kube-proxy/v1.20.11 (linux/amd64) kubernetes/27522a2,client:127.0.0.1 (30-Nov-2021 12:59:14.681) (total time: 24033ms): Trace[1517048506]: [24.033276591s] [24.033276591s] END I1130 12:59:38.715703 1 trace.go:205] Trace[1500240730]: "Get" url:/api/v1/namespaces/default/services/kubernetes,user-agent:kube-apiserver/v1.20.11 (linux/amd64) kubernetes/27522a2,client:::1 (30-Nov-2021 12:59:33.967) (total time: 4748ms): Trace[1500240730]: ---"About to write a response" 4748ms (12:59:00.715) Trace[1500240730]: [4.74865406s] [4.74865406s] END I1130 12:59:38.716295 1 trace.go:205] Trace[74996997]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.20.11 (linux/amd64) kubernetes/27522a2/leader-election,client:127.0.0.1 (30-Nov-2021 12:59:37.988) (total time: 727ms): Trace[74996997]: ---"About to write a response" 727ms (12:59:00.716) Trace[74996997]: [727.374816ms] [727.374816ms] END I1130 12:59:38.717563 1 trace.go:205] Trace[1075975763]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.20.11 (linux/amd64) kubernetes/27522a2/leader-election,client:127.0.0.1 (30-Nov-2021 12:59:30.646) (total time: 8071ms): Trace[1075975763]: ---"About to write a response" 8071ms (12:59:00.717) Trace[1075975763]: [8.071240495s] [8.071240495s] END I1130 12:59:56.024633 1 client.go:360] parsed scheme: "passthrough" I1130 12:59:56.024697 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.1:2379 0 }] } I1130 12:59:56.024707 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 12:59:56.477502 1 client.go:360] parsed scheme: "passthrough" I1130 12:59:56.477563 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.2:2379 0 }] } I1130 12:59:56.477586 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 13:00:00.283077 1 client.go:360] parsed scheme: "passthrough" I1130 13:00:00.283125 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.3:2379 0 }] } I1130 13:00:00.283134 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 13:00:32.320759 1 client.go:360] parsed scheme: "passthrough" I1130 13:00:32.320803 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.1:2379 0 }] } I1130 13:00:32.320814 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 13:00:33.044482 1 client.go:360] parsed scheme: "passthrough" I1130 13:00:33.044533 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.3:2379 0 }] } I1130 13:00:33.044542 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 13:00:37.049272 1 client.go:360] parsed scheme: "passthrough" I1130 13:00:37.049396 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.2:2379 0 }] } I1130 13:00:37.049455 1 clientconn.go:948] ClientConn switching balancer to "pick_first" I1130 13:01:04.742493 1 client.go:360] parsed scheme: "passthrough" I1130 13:01:04.742580 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://10.x.x.3:2379 0 }] } I1130 13:01:04.742591 1 clientconn.go:948] ClientConn switching balancer to "pick_first"

Please run rke remove so all nodes are cleaned (this will remove all cluster data) and run rke --debug up on the cleaned nodes and provide that log, this hopefully shed more light on this issue as it does not reproduce on new nodes with RKE 1.3.1.

Here is the log from rke --debug up after doing rke remove

level=info msg=\"Running RKE version: v1.3.1\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"audit log policy found in cluster.yml\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"Initiating Kubernetes cluster\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Loading data.json from local source\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"data.json SHA256 checksum: 6f17b3853e6e1796d654cd7660bc3470161f34581010127164031acbd89f2fbe\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"No DNS provider configured, setting default based on cluster version [1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"DNS provider set to [coredns]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking if cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"No input provided for maxUnavailableWorker, setting it to default value of 10 percent\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"No input provided for maxUnavailableControlplane, setting it to default value of 1\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking network mode for ingress for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking ingress default backend for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"[state] previous state found, this is not a legacy cluster\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking if cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"Building Kubernetes cluster\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.1]: types.Info{ID:\\"54Z5:I2V3:T26I:N4E6:DQ6U:OCTJ:MH23:LXDX:7H45:YW24:I3VO:O6KF\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.489608957Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa380), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc01\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.2]: types.Info{ID:\\"UPIJ:BH6S:NTAH:QMBK:LPGZ:DA4Y:QML3:XZCQ:AYPF:YH5R:X4YL:3Z36\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.639028819Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa690), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc02\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.3]: types.Info{ID:\\"JN3T:OYKH:B4FY:UZYL:A3CE:HY2N:MZSI:D2PB:GMWM:IGG3:CGXI:4TRM\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.642989055Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa850), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc03\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if cluster version [1.21.4-rancher1-1] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Cluster version [1.21.4-rancher1-1] needs to have kube-api audit log enabled\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.21.4-rancher1-1]\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"[network] No hosts added existing cluster, skipping port check\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:40Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:41Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"\ntime=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"\ntime=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"\ntime=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:46Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:46Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:47Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:47Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:47Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:47Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.3] is [0]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.3]\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:48Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:48Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:49Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:49Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:49Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:49Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.1] is [0]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.1]\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:50Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:50Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:51Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:51Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:51Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:51Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.2] is [0]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Reconciling cluster state\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Check worker hosts to be deleted\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Check Control plane hosts to be deleted\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"\ntime=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Provided value for maxUnavailable: {1 0 10%}\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Parsed value of maxUnavailable: 1\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Provided value for maxUnavailable: {0 1 }\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Parsed value of maxUnavailable: 1\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Encryption is disabled in both current and new spec; no action is required\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Pre-pulling kubernetes images\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Kubernetes images pulled successfully\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"[etcd] Building up etcd plane..\"\ntime=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:52Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:53Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:53Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:53Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:53Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.1] is [0]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.1:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:20:54Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:20:55Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:00Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:00Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:01Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:01Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:01Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.1] is [0]\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:02Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:03Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:03Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:03Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:03Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:04Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:21:04Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:04Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:21:05Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:05Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.2] is [0]\"\ntime=\"2021-12-01T11:21:05Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:05Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:05Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.2:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:11Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:11Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:44Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:44Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:44Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:44Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:21:45Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.2] is [0]\"\ntime=\"2021-12-01T11:21:45Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:45Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:45Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:45Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:46Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:46Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:46Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:46Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:47Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:47Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:47Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:47Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:21:47Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:47Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:21:48Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:48Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:48Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:49Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:49Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:49Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:49Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.3] is [0]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.3:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:50Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:51Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:56Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:56Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:56Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:56Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:56Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"\ntime=\"2021-12-01T11:21:57Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:57Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.3] is [0]\"\ntime=\"2021-12-01T11:21:57Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:57Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:57Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:57Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:57Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:57Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:58Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:58Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:59Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:21:59Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:21:59Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"\ntime=\"2021-12-01T11:21:59Z\" level=debug msg=\"[etcd] check etcd cluster health on host [10.x.x.1]\"\ntime=\"2021-12-01T11:22:00Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"\ntime=\"2021-12-01T11:22:00Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-12-01T11:22:00Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:22:05Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"\ntime=\"2021-12-01T11:22:10Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"\ntime=\"2021-12-01T11:22:15Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"\ntime=\"2021-12-01T11:22:20Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"\ntime=\"2021-12-01T11:22:25Z\" level=debug msg=\"Found node 10.x.x.1 in NotReady state\"\ntime=\"2021-12-01T11:22:25Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-12-01T11:22:25Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:22:30Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #2\"\ntime=\"2021-12-01T11:22:35Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #3\"\ntime=\"2021-12-01T11:22:40Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #4\"\ntime=\"2021-12-01T11:22:45Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #5\"\ntime=\"2021-12-01T11:22:50Z\" level=debug msg=\"Found node 10.x.x.2 in NotReady state\"\ntime=\"2021-12-01T11:22:50Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-12-01T11:22:50Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:22:55Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #2\"\ntime=\"2021-12-01T11:23:00Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #3\"\ntime=\"2021-12-01T11:23:05Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #4\"\ntime=\"2021-12-01T11:23:10Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #5\"\ntime=\"2021-12-01T11:23:15Z\" level=debug msg=\"Found node 10.x.x.3 in NotReady state\"\ntime=\"2021-12-01T11:23:15Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"\ntime=\"2021-12-01T11:23:15Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"\ntime=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:20Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.3]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"\ntime=\"2021-12-01T11:23:20Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.2]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"\ntime=\"2021-12-01T11:23:21Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.1]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"\ntime=\"2021-12-01T11:23:25Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:25Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:25Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10252/healthz for service [kube-controller-manager] on host [10.x.x.3]: Get \\"http://localhost:10252/healthz\\": Unable to access the service on localhost:10252. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:28Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:29Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10251/healthz for service [kube-scheduler] on host [10.x.x.2]: Get \\"http://localhost:10251/healthz\\": Unable to access the service on localhost:10251. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:30Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:30Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:31Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:31Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:31Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:32Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:32Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:33Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"\ntime=\"2021-12-01T11:23:33Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:33Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:33Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:33Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:33Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:33Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:34Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:35Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10251/healthz for service [kube-scheduler] on host [10.x.x.3]: Get \\"http://localhost:10251/healthz\\": Unable to access the service on localhost:10251. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:35Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"\ntime=\"2021-12-01T11:23:35Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:36Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:36Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:40Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"\ntime=\"2021-12-01T11:23:40Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:40Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:40Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:41Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:41Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:41Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:41Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:41Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Building up Worker Plane..\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.1]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.2]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.3]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.1]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.2]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.3]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:55Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:23:56Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"\ntime=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"\ntime=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"\ntime=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"\ntime=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"\ntime=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"\ntime=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"\ntime=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"\ntime=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"\ntime=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"\ntime=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"\ntime=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"\ntime=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"\ntime=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #7\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #7\"\ntime=\"2021-12-01T11:24:29Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.3] is healthy\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:29Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:24:29Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:30Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:24:30Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:30Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:30Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:30Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:24:30Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:30Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.3]\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.2] is healthy\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:24:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.1]\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.2]\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started Worker Plane..\"\ntime=\"2021-12-01T11:24:36Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-12-01T11:24:36Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:24:41Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"\ntime=\"2021-12-01T11:24:46Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"\ntime=\"2021-12-01T11:24:51Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"\ntime=\"2021-12-01T11:24:56Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"\ntime=\"2021-12-01T11:25:01Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"\ntime=\"2021-12-01T11:25:01Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"\ntime=\"2021-12-01T11:25:01Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #1\"\ntime=\"2021-12-01T11:25:06Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #2\"\ntime=\"2021-12-01T11:25:11Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #3\"\ntime=\"2021-12-01T11:25:16Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #4\"\ntime=\"2021-12-01T11:25:21Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #5\"\ntime=\"2021-12-01T11:25:26Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"\ntime=\"2021-12-01T11:25:26Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"\ntime=\"2021-12-01T11:25:26Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #1\"\ntime=\"2021-12-01T11:25:31Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #2\"\ntime=\"2021-12-01T11:25:36Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #3\"\ntime=\"2021-12-01T11:25:41Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #4\"\ntime=\"2021-12-01T11:25:46Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #5\"\ntime=\"2021-12-01T11:25:51Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"\ntime=\"2021-12-01T11:25:51Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"\ntime=\"2021-12-01T11:25:51Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"\ntime=\"2021-12-01T11:25:51Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"\ntime=\"2021-12-01T11:25:51Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"\ntime=\"2021-12-01T11:25:56Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"\ntime=\"2021-12-01T11:26:01Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"\ntime=\"2021-12-01T11:26:06Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"\ntime=\"2021-12-01T11:26:11Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"\ntime=\"2021-12-01T11:26:16Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"\ntime=\"2021-12-01T11:26:16Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\"", "stdout_lines": ["time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Loglevel set to [debug]\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"Running RKE version: v1.3.1\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"audit log policy found in cluster.yml\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"Initiating Kubernetes cluster\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Loading data.json from local source\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"data.json SHA256 checksum: 6f17b3853e6e1796d654cd7660bc3470161f34581010127164031acbd89f2fbe\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"No DNS provider configured, setting default based on cluster version [1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"DNS provider set to [coredns]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking if cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"No input provided for maxUnavailableWorker, setting it to default value of 10 percent\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"No input provided for maxUnavailableControlplane, setting it to default value of 1\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking network mode for ingress for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking ingress default backend for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"[state] previous state found, this is not a legacy cluster\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"Successfully Deployed state file at [./rancher-cluster.rkestate]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Checking if cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Cluster version [1.20.11-rancher1-2] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"Building Kubernetes cluster\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.1]\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.2]\"", "time=\"2021-12-01T11:20:39Z\" level=info msg=\"[dialer] Setup tunnel for host [10.x.x.3]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.3]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.1]\"", "time=\"2021-12-01T11:20:39Z\" level=debug msg=\"Connecting to Docker API for host [10.x.x.2]\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.1]: types.Info{ID:\\"54Z5:I2V3:T26I:N4E6:DQ6U:OCTJ:MH23:LXDX:7H45:YW24:I3VO:O6KF\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.489608957Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa380), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc01\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.2]: types.Info{ID:\\"UPIJ:BH6S:NTAH:QMBK:LPGZ:DA4Y:QML3:XZCQ:AYPF:YH5R:X4YL:3Z36\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.639028819Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa690), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc02\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Docker Info found for host [10.x.x.3]: types.Info{ID:\\"JN3T:OYKH:B4FY:UZYL:A3CE:HY2N:MZSI:D2PB:GMWM:IGG3:CGXI:4TRM\\", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:7, Driver:\\"overlay2\\", DriverStatus:[][2]string{[2]string{\\"Backing Filesystem\\", \\"extfs\\"}, [2]string{\\"Supports d_type\\", \\"true\\"}, [2]string{\\"Native Overlay Diff\\", \\"true\\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\\"local\\"}, Network:[]string{\\"bridge\\", \\"host\\", \\"ipvlan\\", \\"macvlan\\", \\"null\\", \\"overlay\\"}, Authorization:[]string(nil), Log:[]string{\\"awslogs\\", \\"fluentd\\", \\"gcplogs\\", \\"gelf\\", \\"journald\\", \\"json-file\\", \\"local\\", \\"logentries\\", \\"splunk\\", \\"syslog\\"}}, MemoryLimit:true, SwapLimit:false, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:24, OomKillDisable:true, NGoroutines:35, SystemTime:\\"2021-12-01T11:20:40.642989055Z\\", LoggingDriver:\\"json-file\\", CgroupDriver:\\"cgroupfs\\", CgroupVersion:\\"\\", NEventsListener:0, KernelVersion:\\"4.15.0-88-generic\\", OperatingSystem:\\"Ubuntu 18.04.4 LTS\\", OSVersion:\\"\\", OSType:\\"linux\\", Architecture:\\"x86_64\\", IndexServerAddress:\\"https://index.docker.io/v1/\\", RegistryConfig:(registry.ServiceConfig)(0xc0000fa850), NCPU:4, MemTotal:16819859456, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\\"/var/lib/docker\\", HTTPProxy:\\"\\", HTTPSProxy:\\"\\", NoProxy:\\"\\", Name:\\"ausdvshazzrnc03\\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\\"19.03.6\\", ClusterStore:\\"\\", ClusterAdvertise:\\"\\", Runtimes:map[string]types.Runtime{\\"runc\\":types.Runtime{Path:\\"runc\\", Args:[]string(nil), Shim:(types.ShimConfig)(nil)}}, DefaultRuntime:\\"runc\\", Swarm:swarm.Info{NodeID:\\"\\", NodeAddr:\\"\\", LocalNodeState:\\"inactive\\", ControlAvailable:false, Error:\\"\\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\\"\\", InitBinary:\\"docker-init\\", ContainerdCommit:types.Commit{ID:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\", Expected:\\"7b11cfaabd73bb80907dd23182b9347b4245eb5d\\"}, RuncCommit:types.Commit{ID:\\"v1.0.2-0-g52b36a2\\", Expected:\\"v1.0.2-0-g52b36a2\\"}, InitCommit:types.Commit{ID:\\"fec3683\\", Expected:\\"fec3683\\"}, SecurityOptions:[]string{\\"name=apparmor\\", \\"name=seccomp,profile=default\\"}, ProductLicense:\\"\\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string{\\"WARNING: No swap limit support\\"}}\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if cluster version [1.21.4-rancher1-1] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Cluster version [1.21.4-rancher1-1] needs to have kube-api audit log enabled\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Enabling kube-api audit log for cluster version [v1.21.4-rancher1-1]\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: controlplane\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: worker\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.1 has role: etcd\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: controlplane\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: worker\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.2 has role: etcd\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: controlplane\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: worker\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Host: 10.x.x.3 has role: etcd\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"[network] No hosts added existing cluster, skipping port check\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:40Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:41Z\" level=info msg=\"Starting container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"", "time=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"", "time=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:41Z\" level=debug msg=\"[certificates] Successfully started Certificate deployer container: cert-deployer\"", "time=\"2021-12-01T11:20:41Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Checking if container [cert-deployer] is running on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Removing container [cert-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:46Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.3]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:46Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:46Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:47Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:47Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:47Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:47Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.3] is [0]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.3]\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.1]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:48Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:48Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:49Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:49Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:49Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:49Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.1] is [0]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.1]\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.x.x.2]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"[remove/file-deployer] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:50Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:50Z\" level=info msg=\"Starting container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:51Z\" level=info msg=\"Successfully started [file-deployer] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:51Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:51Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:51Z\" level=info msg=\"Container [file-deployer] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Waiting for [file-deployer] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Exit code for [file-deployer] container on host [10.x.x.2] is [0]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[remove/file-deployer] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[remove/file-deployer] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Removing container [file-deployer] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Reconciling cluster state\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Check etcd hosts to be added\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Check worker hosts to be deleted\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Check Control plane hosts to be deleted\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.1] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.2] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[reconcile] Rebuilding and updating local kube config, creating new kubeconfig\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Deploying admin Kubeconfig locally at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Successfully Deployed local admin kubeconfig at [./kube_config_rancher-cluster.yml]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Using ./kube_config_rancher-cluster.yml to connect to Kubernetes cluster..\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"[version] Getting Kubernetes server version..\"", "time=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] host [10.x.x.3] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2021-12-01T11:20:52Z\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[reconcile] Reconciled cluster state successfully\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Provided value for maxUnavailable: {1 0 10%}\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"max_unavailable_worker got rounded down to 0, resetting to 1\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Parsed value of maxUnavailable: 1\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Provided value for maxUnavailable: {0 1 }\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Parsed value of maxUnavailable: 1\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Setting maxUnavailable for worker nodes to: 1\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Setting maxUnavailable for controlplane nodes to: 1\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Encryption is disabled in both current and new spec; no action is required\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Pre-pulling kubernetes images\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Kubernetes images pulled successfully\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"[etcd] Building up etcd plane..\"", "time=\"2021-12-01T11:20:52Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:52Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:53Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:53Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:53Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:53Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.1] is [0]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.1:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:20:54Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:20:55Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:00Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:00Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:00Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:01Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:01Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:01Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.1]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.1] is [0]\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:02Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:03Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:03Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:03Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:03Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:04Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.1]\"", "time=\"2021-12-01T11:21:04Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:04Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:21:05Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:05Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.2] is [0]\"", "time=\"2021-12-01T11:21:05Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:05Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:05Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.2:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:06Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:11Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:11Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:44Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:44Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:44Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:44Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.2]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:21:45Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.2] is [0]\"", "time=\"2021-12-01T11:21:45Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:45Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:45Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:45Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:46Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:46Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:46Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:46Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:47Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:47Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:47Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:47Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:21:47Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:47Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.2]\"", "time=\"2021-12-01T11:21:48Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:48Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:48Z\" level=info msg=\"Starting container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:49Z\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:49Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:49Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:49Z\" level=info msg=\"Container [etcd-fix-perm] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"Exit code for [etcd-fix-perm] container on host [10.x.x.3] is [0]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-fix-perm] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-fix-perm] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Removing container [etcd-fix-perm] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"Checking if image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Starting container [etcd] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"[etcd] Successfully started [etcd] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"Extracted version [v0.1.78] from image [rancher/rke-tools:v0.1.78]\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.x.x.3:2379 --retention=24h --creation=6h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"[remove/etcd-rolling-snapshots] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:50Z\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:51Z\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:56Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:56Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:56Z\" level=info msg=\"Starting container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:56Z\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:56Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:56Z\" level=info msg=\"Container [rke-bundle-cert] is still running on host [10.x.x.3]: stderr: [], stdout: []\"", "time=\"2021-12-01T11:21:57Z\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:57Z\" level=debug msg=\"Exit code for [rke-bundle-cert] container on host [10.x.x.3] is [0]\"", "time=\"2021-12-01T11:21:57Z\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:57Z\" level=info msg=\"Removing container [rke-bundle-cert] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:57Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:57Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:57Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:57Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:57Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:58Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:58Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[etcd] Creating log link for Container [etcd] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:58Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:58Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:59Z\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:21:59Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=debug msg=\"[etcd] Successfully created log link for Container [etcd] on host [10.x.x.3]\"", "time=\"2021-12-01T11:21:59Z\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"", "time=\"2021-12-01T11:21:59Z\" level=debug msg=\"[etcd] check etcd cluster health on host [10.x.x.1]\"", "time=\"2021-12-01T11:22:00Z\" level=info msg=\"[etcd] etcd host [10.x.x.1] reported healthy=true\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.11-rancher1-2]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v1.20.11-rancher1] from image [rancher/hyperkube:v1.20.11-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Version [3.4.15-rancher1] is equal or higher than version [3.2.99]\"", "time=\"2021-12-01T11:22:00Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-12-01T11:22:00Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"", "time=\"2021-12-01T11:22:05Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"", "time=\"2021-12-01T11:22:10Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"", "time=\"2021-12-01T11:22:15Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"", "time=\"2021-12-01T11:22:20Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"", "time=\"2021-12-01T11:22:25Z\" level=debug msg=\"Found node 10.x.x.1 in NotReady state\"", "time=\"2021-12-01T11:22:25Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-12-01T11:22:25Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #1\"", "time=\"2021-12-01T11:22:30Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #2\"", "time=\"2021-12-01T11:22:35Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #3\"", "time=\"2021-12-01T11:22:40Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #4\"", "time=\"2021-12-01T11:22:45Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #5\"", "time=\"2021-12-01T11:22:50Z\" level=debug msg=\"Found node 10.x.x.2 in NotReady state\"", "time=\"2021-12-01T11:22:50Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-12-01T11:22:50Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #1\"", "time=\"2021-12-01T11:22:55Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #2\"", "time=\"2021-12-01T11:23:00Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #3\"", "time=\"2021-12-01T11:23:05Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #4\"", "time=\"2021-12-01T11:23:10Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #5\"", "time=\"2021-12-01T11:23:15Z\" level=debug msg=\"Found node 10.x.x.3 in NotReady state\"", "time=\"2021-12-01T11:23:15Z\" level=info msg=\"Attempting upgrade of controlplane components on following hosts in NotReady status: 10.x.x.1,10.x.x.2,10.x.x.3\"", "time=\"2021-12-01T11:23:15Z\" level=info msg=\"[controlplane] Building up Controller Plane..\"", "time=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:15Z\" level=debug msg=\"[remove/nginx-proxy] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"[remove/nginx-proxy] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"Starting container [kube-apiserver] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:16Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:20Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.3]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"", "time=\"2021-12-01T11:23:20Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.2]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"", "time=\"2021-12-01T11:23:21Z\" level=debug msg=\"[healthcheck] Service [kube-apiserver] is not healthy on host [10.x.x.1]. Response code: [403], response body: {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"forbidden: User \\\\"kube-apiserver\\\\" cannot get path \\\\"/healthz\\\\"\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{},\\"code\\":403}\n, try #1\"", "time=\"2021-12-01T11:23:25Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.3] is healthy\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:25Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.2] is healthy\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:25Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:25Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [10.x.x.1] is healthy\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-apiserver] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:26Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-apiserver] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"Starting container [kube-controller-manager] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:27Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10252/healthz for service [kube-controller-manager] on host [10.x.x.3]: Get \\"http://localhost:10252/healthz\\": Unable to access the service on localhost:10252. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:27Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.2] is healthy\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.1] is healthy\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:28Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:29Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10251/healthz for service [kube-scheduler] on host [10.x.x.2]: Get \\"http://localhost:10251/healthz\\": Unable to access the service on localhost:10251. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:30Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:30Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.1] is healthy\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:31Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:31Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:31Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:32Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:32Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:33Z\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [10.x.x.3] is healthy\"", "time=\"2021-12-01T11:23:33Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-controller-manager] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:33Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:33Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:33Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:33Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:33Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-controller-manager] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"Starting container [kube-scheduler] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:34Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:35Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10251/healthz for service [kube-scheduler] on host [10.x.x.3]: Get \\"http://localhost:10251/healthz\\": Unable to access the service on localhost:10251. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:35Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.2] is healthy\"", "time=\"2021-12-01T11:23:35Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:36Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:36Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:40Z\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [10.x.x.3] is healthy\"", "time=\"2021-12-01T11:23:40Z\" level=debug msg=\"[controlplane] Creating log link for Container [kube-scheduler] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:40Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:40Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:40Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:40Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:41Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:41Z\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:41Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:41Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:41Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[controlplane] Successfully created log link for Container [kube-scheduler] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Building up Worker Plane..\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Checking if container [service-sidekick] is running on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Checking if container [service-sidekick] is eligible for upgrade on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[sidekick] Container [service-sidekick] is not eligible for upgrade on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[sidekick] Sidekick container already created on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"Starting container [kubelet] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[worker] Successfully started [kubelet] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kubelet] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.1]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.2]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:42Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.3]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.1]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.2]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:23:48Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10248/healthz for service [kubelet] on host [10.x.x.3]: Get \\"http://localhost:10248/healthz\\": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.1] is healthy\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.2] is healthy\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"[healthcheck] service [kubelet] on host [10.x.x.3] is healthy\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[worker] Creating log link for Container [kubelet] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:54Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[worker] Successfully created log link for Container [kubelet] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"Checking if image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Image [rancher/hyperkube:v1.20.11-rancher1] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.1]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.2]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"Starting container [kube-proxy] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[worker] Successfully started [kube-proxy] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.x.x.3]\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:55Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:23:56Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #1\"", "time=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:24:01Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #2\"", "time=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"", "time=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"", "time=\"2021-12-01T11:24:07Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #3\"", "time=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"", "time=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"", "time=\"2021-12-01T11:24:12Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #4\"", "time=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"", "time=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"", "time=\"2021-12-01T11:24:18Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #5\"", "time=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"", "time=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"", "time=\"2021-12-01T11:24:24Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.3]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #6\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.1]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #7\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"[healthcheck] Failed to check http://localhost:10256/healthz for service [kube-proxy] on host [10.x.x.2]: Get \\"http://localhost:10256/healthz\\": Unable to access the service on localhost:10256. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), try #7\"", "time=\"2021-12-01T11:24:29Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.3] is healthy\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:29Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:24:29Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:30Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:24:30Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:30Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:30Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:30Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.3], try #1\"", "time=\"2021-12-01T11:24:30Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:30Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.3]\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.1] is healthy\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"[healthcheck] service [kube-proxy] on host [10.x.x.2] is healthy\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[worker] Creating log link for Container [kube-proxy] on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"[remove/rke-log-linker] Container doesn't exist on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:35Z\" level=debug msg=\"Checking if image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"Image [rancher/rke-tools:v0.1.78] exists on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:24:35Z\" level=info msg=\"Starting container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.1], try #1\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started [rke-log-linker] container on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Checking if container is running on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[remove/rke-log-linker] Removing container on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"Removing container [rke-log-linker] on host [10.x.x.2], try #1\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.1]\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"[worker] Successfully created log link for Container [kube-proxy] on host [10.x.x.2]\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[worker] Successfully started Worker Plane..\"", "time=\"2021-12-01T11:24:36Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-12-01T11:24:36Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"", "time=\"2021-12-01T11:24:41Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"", "time=\"2021-12-01T11:24:46Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"", "time=\"2021-12-01T11:24:51Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"", "time=\"2021-12-01T11:24:56Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"", "time=\"2021-12-01T11:25:01Z\" level=error msg=\"Host 10.x.x.1 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found\"", "time=\"2021-12-01T11:25:01Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.2, try #1\"", "time=\"2021-12-01T11:25:01Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #1\"", "time=\"2021-12-01T11:25:06Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #2\"", "time=\"2021-12-01T11:25:11Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #3\"", "time=\"2021-12-01T11:25:16Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #4\"", "time=\"2021-12-01T11:25:21Z\" level=debug msg=\"Checking node list for node [10.x.x.2], try #5\"", "time=\"2021-12-01T11:25:26Z\" level=error msg=\"Host 10.x.x.2 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.2: \\"10.x.x.2\\" not found\"", "time=\"2021-12-01T11:25:26Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.3, try #1\"", "time=\"2021-12-01T11:25:26Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #1\"", "time=\"2021-12-01T11:25:31Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #2\"", "time=\"2021-12-01T11:25:36Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #3\"", "time=\"2021-12-01T11:25:41Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #4\"", "time=\"2021-12-01T11:25:46Z\" level=debug msg=\"Checking node list for node [10.x.x.3], try #5\"", "time=\"2021-12-01T11:25:51Z\" level=error msg=\"Host 10.x.x.3 failed to report Ready status with error: [controlplane] Error getting node 10.x.x.3: \\"10.x.x.3\\" not found\"", "time=\"2021-12-01T11:25:51Z\" level=info msg=\"[controlplane] Processing controlplane hosts for upgrade 1 at a time\"", "time=\"2021-12-01T11:25:51Z\" level=info msg=\"Processing controlplane host 10.x.x.1\"", "time=\"2021-12-01T11:25:51Z\" level=info msg=\"[controlplane] Now checking status of node 10.x.x.1, try #1\"", "time=\"2021-12-01T11:25:51Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #1\"", "time=\"2021-12-01T11:25:56Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #2\"", "time=\"2021-12-01T11:26:01Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #3\"", "time=\"2021-12-01T11:26:06Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #4\"", "time=\"2021-12-01T11:26:11Z\" level=debug msg=\"Checking node list for node [10.x.x.1], try #5\"", "time=\"2021-12-01T11:26:16Z\" level=error msg=\"Failed to upgrade hosts: 10.x.x.1 with error [[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]\"", "time=\"2021-12-01T11:26:16Z\" level=fatal msg=\"[controlPlane] Failed to upgrade Control Plane: [[[controlplane] Error getting node 10.x.x.1: \\"10.x.x.1\\" not found]]\""]}

rke remove cleans the nodes and removes any files that have old state like cluster.rkestate and the kubeconfig generated. The log line level=debug msg="[state] previous state found, this is not a legacy cluster"\ntime="2021-12-01T11:20:39Z" indicates this has not happened, this is a bug on its own as it needs to remove those files so if you can share the before and after output of ls -la of the directory where you are running rke remove and rke up, we can look at that, if those are then removed, the output of rke up is what we are looking for to see why it can't create the cluster properly.

I did another RKE remove and redid rke up. I see the network pods are coming up. Not sure how the rke cluster state file was cache. RKE up is run from a docker container and the cluster file is created within that container. After the task completes the container is deleted. So not sure where and how the cluster state file was cached. The VMs were also recreated to make sure to be used as fresh slate.

RKE up succeeds but the etcd server times out. I will raise another issue for it. Closing this issue. Thank you @superseb

Using system_images is no longer supported, please remove it from the YAML and start using kubernetes_version

Hi @superseb! Sorry for asking you here, but is this is applicable for rke up? rke config still creates this section in YAML and that section looks up-to-date. There is a couple of issues here also related to system_images and kubernetes_version`:

Another concern is how are we supposed to know which images to collect for air-gapped environments?

The docs were changed to move away from system_images and exclusively use kubernetes_version (also described on https://rke.docs.rancher.com/upgrades).

rke config has not been modified according to this change, and until this is corrected, manual creation of cluster.yml is recommended.

Listing the images needed for a Kubernetes version can be done using rke config -s -a.

Wow, fantastic, thank you very much indeed!

rancher / rke

RKE network plugin containers dont start #2756