Closed JasinlikeEatingOrange closed 2 years ago
DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:26.861374 I | embed: rejected connection from "10.109.31.113:52870" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:55399->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:26.890365 I | embed: rejected connection from "10.109.31.113:52872" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:60436->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:27.642290 I | embed: rejected connection from "10.109.31.113:58726" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:55485->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:27.707116 I | embed: rejected connection from "10.109.31.113:58728" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:51638->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"])
it is docker logs
Is this a clean install or an upgrade? If clean install, were the nodes cleaned properly before (re)-using them?
Is this a clean install or an upgrade? If clean install, were the nodes cleaned properly before (re)-using them?
it is clean install ,i use this commands:
105 docker stop $(docker ps -a )
106 docker stop $(docker ps -aq )
107 docker system --help
108 docker system prune -f
109 docker volume rm $(docker volume ls -q)
110 docker image rm $(docker image ls -q)
111 rm -rf /etc/ceph /etc/cni /etc/kubernetes /opt/cni /opt/rke
/run/secrets/kubernetes.io /run/calico /run/flannel /var/lib/calico
/var/lib/etcd /var/lib/cni /var/lib/kubelet /var/lib/rancher/rke/log
/var/log/containers /var/log/pods /var/run/calico
112 reboot
then finished i login my user "rke" and rke up,it just same problem, how sholud i need to clean and reinstall ?
rke remove
will clean everything as well, did you clean up local files created by RKE as well? (cluster.rkestate
and kube_config_cluster.yml
) ?
rke remove
will clean everything as well, did you clean up local files created by RKE as well? (cluster.rkestate
andkube_config_cluster.yml
) ?
i did not clean these two files . let me have a try. thx a lot!!
rke remove use
rke remove
it will auto delcluster.rkestate
andkube_config_cluster.yml
. but it throw same prblems.host [10.109.31.135] failed to check etcd health: failed to get /health for host [10.109.31.135]: Get "https://10.109.31.135:2379/health": net/http: TLS handshake timeout
Please provide the full log of rke --debug up
. Does it work with a normal release (non rc like v1.2.9)?
Please provide the full log of
rke --debug up
. Does it work with a normal release (non rc like v1.2.9)?
2021-07-14 10:00:42.236397 I | embed: rejected connection from "10.109.31.113:51426" (error "remote error: tls: bad certificate", ServerName "")
there is a new error in my docker logs
time="2021-07-14T18:16:48+08:00" level=debug msg="Loglevel set to [debug]"
time="2021-07-14T18:16:48+08:00" level=warning msg="This is not an officially supported version (v1.3.0-rc8) of RKE. Please download the latest official release at https://github.com/rancher/rke/releases"
time="2021-07-14T18:16:48+08:00" level=info msg="Running RKE version: v1.3.0-rc8"
time="2021-07-14T18:16:48+08:00" level=debug msg="audit log policy found in cluster.yml"
time="2021-07-14T18:16:48+08:00" level=info msg="Initiating Kubernetes cluster"
time="2021-07-14T18:16:48+08:00" level=debug msg="Loading data.json from local source"
time="2021-07-14T18:16:48+08:00" level=debug msg="data.json SHA256 checksum: 705b1e1058c996d5bab9ec46a6b8066fe357a22978999a44faffe1e70638c5e3"
time="2021-07-14T18:16:48+08:00" level=debug msg="No DNS provider configured, setting default based on cluster version [1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="DNS provider set to [coredns]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Enabling kube-api audit log for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="No input provided for maxUnavailableWorker, setting it to default value of 10 percent"
time="2021-07-14T18:16:48+08:00" level=debug msg="No input provided for maxUnavailableControlplane, setting it to default value of 1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking ingress default backend for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="[state] previous state found, this is not a legacy cluster"
time="2021-07-14T18:16:48+08:00" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
time="2021-07-14T18:16:48+08:00" level=info msg="[certificates] Generating admin certificates and kubeconfig"
time="2021-07-14T18:16:48+08:00" level=info msg="Successfully Deployed state file at [./cluster.rkestate]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Enabling kube-api audit log for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=info msg="Building Kubernetes cluster"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.135]: types.Info{ID:\"7PZW:ENLM:7ZC2:KKWZ:CNEI:GCKI:EHGM:NP2N:HWNQ:ANEQ:2O2W:DTGP\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.417684777+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc0003a83f0), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyprk1\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.137]: types.Info{ID:\"T4WJ:2Y77:ZWUX:BWC5:DQDP:DPBW:ZO2X:U24U:6E2Q:PS2D:KZT5:AH4O\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.466766633+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc000246070), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyp2\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.136]: types.Info{ID:\"5Y53:KY6C:RART:JOV2:USAO:YIPT:HBME:PJGF:67O3:36YC:55M5:5JM7\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.473340071+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc000246230), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyp1\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=info msg="[network] Deploying port listener containers"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.137]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Port listener containers deployed successfully"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Running etcd <-> etcd port checks"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:54+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Running control plane -> etcd port checks"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Running control plane -> worker port checks"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Running workers -> control plane port checks"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Checking KubeAPI port Control Plane hosts"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Removing port listener containers"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="[network] Port listener containers removed successfully"
time="2021-07-14T18:17:00+08:00" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.135] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.136] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.137] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
time="2021-07-14T18:17:07+08:00" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
time="2021-07-14T18:17:07+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:08+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:09+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:09+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:09+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:10+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:11+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:12+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:12+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:12+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:13+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:14+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
time="2021-07-14T18:17:14+08:00" level=info msg="[reconcile] Reconciling cluster state"
time="2021-07-14T18:17:14+08:00" level=info msg="[reconcile] This is newly generated cluster"
time="2021-07-14T18:17:14+08:00" level=debug msg="Encryption is disabled in both current and new spec; no action is required"
time="2021-07-14T18:17:14+08:00" level=info msg="Pre-pulling kubernetes images"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.135]"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.136]"
time="2021-07-14T18:17:14+08:00" level=info msg="Kubernetes images pulled successfully"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=info msg="[etcd] Building up etcd plane.."
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:15+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:16+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:16+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:16+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.135:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:17+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:17+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:17+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:17+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:18+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.135]"
time="2021-07-14T18:17:23+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:23+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:23+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:24+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.135]"
time="2021-07-14T18:17:24+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:24+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:25+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:25+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:25+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:25+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:28+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:29+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:30+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:30+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.136:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:31+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:31+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.136]"
time="2021-07-14T18:17:36+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:36+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:37+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:38+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:38+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:38+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:38+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:39+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:40+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:40+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:42+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:43+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:44+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:44+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.137:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:45+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:45+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.137]"
time="2021-07-14T18:17:50+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:50+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:51+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:52+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:52+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:52+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:52+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:53+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:55+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:55+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
time="2021-07-14T18:17:55+08:00" level=debug msg="[etcd] check etcd cluster health on host [10.109.31.135]"
time="2021-07-14T18:18:05+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:21+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:36+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:51+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:06+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:21+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:36+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:51+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:20:06+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
Please provide the log after running rke remove
, this line indicates you still have local state and this is not a fresh/new run:
time="2021-07-14T18:16:48+08:00" level=debug msg="[state] previous state found, this is not a legacy cluster"
This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
RKE version: 1.3.0-rc8
Docker version: (
docker version
,docker info
preferred)20.10.7Operating system and kernel: (
cat /etc/os-release
,uname -r
preferred)centos 7Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)DO
cluster.yml file:
If you intended to deploy Kubernetes in an air-gapped environment,
please consult the documentation on how to configure custom RKE images.
nodes:
Steps to Reproduce: rke up failed Results: [etcd] host [10.109.31.135] failed to check etcd health: failed to get /health for host [10.109.31.135]: Get "https://10.109.31.135:2379/health": net/http: TLS handshake timeout [etcd] host [10.109.31.136] failed to check etcd health: failed to get /health for host [10.109.31.136]: Get "https://10.109.31.136:2379/health": net/http: TLS handshake timeout [etcd] host [10.109.31.137] failed to check etcd health: failed to get /health for host [10.109.31.137]: Get "https://10.109.31.137:2379/health": net/http: TLS handshake timeout