JasinlikeEatingOrange commented 3 years ago

RKE version: 1.3.0-rc8

Docker version: (docker version,docker info preferred)20.10.7

Operating system and kernel: (cat /etc/os-release, uname -r preferred)centos 7

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)DO

cluster.yml file:

If you intended to deploy Kubernetes in an air-gapped environment,

please consult the documentation on how to configure custom RKE images.

nodes:

address: 10.109.31.135 port: "22" internal_address: "" role:
- controlplane
- worker
- etcd hostname_override: "" user: rke docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: []
address: 10.109.31.136 port: "22" internal_address: "" role:
- controlplane
- worker
- etcd hostname_override: "" user: rke docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: []
address: 10.109.31.137 port: "22" internal_address: "" role:
- controlplane
- worker
- etcd hostname_override: "" user: rke docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: []
address: 10.109.31.138 port: "22" internal_address: "" role:
- controlplane
- worker
- etcd hostname_override: "" user: rke docker_socket: /var/run/docker.sock ssh_key: "" ssh_key_path: ~/.ssh/id_rsa ssh_cert: "" ssh_cert_path: "" labels: {} taints: [] services: etcd: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] external_urls: [] ca_cert: "" cert: "" key: "" path: "" uid: 0 gid: 0 snapshot: null retention: "" creation: "" backup_config: null kube-api: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] service_cluster_ip_range: 10.43.0.0/16 service_node_port_range: "" pod_security_policy: false always_pull_images: false secrets_encryption_config: null audit_log: null admission_configuration: null event_rate_limit: null kube-controller: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] cluster_cidr: 10.42.0.0/16 service_cluster_ip_range: 10.43.0.0/16 scheduler: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] kubelet: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] cluster_domain: cluster.local infra_container_image: "" cluster_dns_server: 10.43.0.10 fail_swap_on: false generate_serving_certificate: false kubeproxy: image: "" extra_args: {} extra_binds: [] extra_env: [] win_extra_args: {} win_extra_binds: [] win_extra_env: [] network: plugin: calico options: {} mtu: 0 node_selector: {} update_strategy: null tolerations: [] authentication: strategy: x509 sans: [] webhook: null addons: "" addons_include:
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
https://gist.githubusercontent.com/superseb/499f2caa2637c404af41cfb7e5f4a938/raw/930841ac00653fdff8beca61dab9a20bb8983782/k8s-dashboard-user.yml system_images: etcd: rancher/mirrored-coreos-etcd:v3.4.15-rancher1 alpine: rancher/rke-tools:v0.1.76 nginx_proxy: rancher/rke-tools:v0.1.76 cert_downloader: rancher/rke-tools:v0.1.76 kubernetes_services_sidecar: rancher/rke-tools:v0.1.76 kubedns: rancher/mirrored-k8s-dns-kube-dns:1.15.10 dnsmasq: rancher/mirrored-k8s-dns-dnsmasq-nanny:1.15.10 kubedns_sidecar: rancher/mirrored-k8s-dns-sidecar:1.15.10 kubedns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.1 coredns: rancher/mirrored-coredns-coredns:1.8.0 coredns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.1 nodelocal: rancher/mirrored-k8s-dns-node-cache:1.15.13 kubernetes: rancher/hyperkube:v1.20.7-rancher1 flannel: rancher/coreos-flannel:v0.13.0-rancher1 flannel_cni: rancher/flannel-cni:v0.3.0-rancher6 calico_node: rancher/mirrored-calico-node:v3.17.2 calico_cni: rancher/mirrored-calico-cni:v3.17.2 calico_controllers: rancher/mirrored-calico-kube-controllers:v3.17.2 calico_ctl: rancher/mirrored-calico-ctl:v3.17.2 calico_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.17.2 canal_node: rancher/mirrored-calico-node:v3.17.2 canal_cni: rancher/mirrored-calico-cni:v3.17.2 canal_controllers: rancher/mirrored-calico-kube-controllers:v3.17.2 canal_flannel: rancher/coreos-flannel:v0.13.0-rancher1 canal_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.17.2 weave_node: weaveworks/weave-kube:2.8.1 weave_cni: weaveworks/weave-npc:2.8.1 pod_infra_container: rancher/mirrored-pause:3.2 ingress: rancher/nginx-ingress-controller:nginx-0.43.0-rancher3 ingress_backend: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1 metrics_server: rancher/mirrored-metrics-server:v0.4.1 windows_pod_infra_container: rancher/kubelet-pause:v0.1.6 aci_cni_deploy_container: noiro/cnideploy:5.1.1.0.1ae238a aci_host_container: noiro/aci-containers-host:5.1.1.0.1ae238a aci_opflex_container: noiro/opflex:5.1.1.0.1ae238a aci_mcast_container: noiro/opflex:5.1.1.0.1ae238a aci_ovs_container: noiro/openvswitch:5.1.1.0.1ae238a aci_controller_container: noiro/aci-containers-controller:5.1.1.0.1ae238a aci_gbp_server_container: noiro/gbp-server:5.1.1.0.1ae238a aci_opflex_server_container: noiro/opflex-server:5.1.1.0.1ae238a ssh_key_path: ~/.ssh/id_rsa ssh_cert_path: "" ssh_agent_auth: false authorization: mode: rbac options: {} ignore_docker_version: null enable_cri_dockerd: null kubernetes_version: "" private_registries: [] ingress: provider: "" options: {} node_selector: {} extra_args: {} dns_policy: "" extra_envs: [] extra_volumes: [] extra_volume_mounts: [] update_strategy: null http_port: 0 https_port: 0 network_mode: "" tolerations: [] default_backend: null default_http_backend_priority_class_name: "" nginx_ingress_controller_priority_class_name: "" cluster_name: "" cloud_provider: name: "" prefix_path: "" win_prefix_path: "" addon_job_timeout: 600 bastion_host: address: "" port: "" user: "" ssh_key: "" ssh_key_path: "" ssh_cert: "" ssh_cert_path: "" ignore_proxy_env_vars: false monitoring: provider: "" options: {} node_selector: {} update_strategy: null replicas: null tolerations: [] metrics_server_priority_class_name: "" restore: restore: false snapshot_name: "" rotate_encryption_key: false dns: provider: coredns

Steps to Reproduce: rke up failed Results: [etcd] host [10.109.31.135] failed to check etcd health: failed to get /health for host [10.109.31.135]: Get "https://10.109.31.135:2379/health": net/http: TLS handshake timeout [etcd] host [10.109.31.136] failed to check etcd health: failed to get /health for host [10.109.31.136]: Get "https://10.109.31.136:2379/health": net/http: TLS handshake timeout [etcd] host [10.109.31.137] failed to check etcd health: failed to get /health for host [10.109.31.137]: Get "https://10.109.31.137:2379/health": net/http: TLS handshake timeout

JasinlikeEatingOrange commented 3 years ago

DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:26.861374 I | embed: rejected connection from "10.109.31.113:52870" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:55399->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:26.890365 I | embed: rejected connection from "10.109.31.113:52872" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:60436->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:27.642290 I | embed: rejected connection from "10.109.31.113:58726" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:55485->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"]) 2021-07-14 08:10:27.707116 I | embed: rejected connection from "10.109.31.113:58728" (error "tls: \"10.109.31.113\" does not match any of DNSNames [\"localhost\" \"kubernetes\" \"kubernetes.default\" \"kubernetes.default.svc\" \"kubernetes.default.svc.cluster.local\"] (lookup kubernetes.default.svc.cluster.local on 10.109.31.1:53: read udp 10.109.31.135:51638->10.109.31.1:53: i/o timeout)", ServerName "", IPAddresses ["10.109.31.135" "10.109.31.136" "10.109.31.137" "10.109.31.138" "127.0.0.1" "10.43.0.1"], DNSNames ["localhost" "kubernetes" "kubernetes.default" "kubernetes.default.svc" "kubernetes.default.svc.cluster.local"])

it is docker logs

superseb commented 3 years ago

Is this a clean install or an upgrade? If clean install, were the nodes cleaned properly before (re)-using them?

JasinlikeEatingOrange commented 3 years ago

Is this a clean install or an upgrade? If clean install, were the nodes cleaned properly before (re)-using them?

it is clean install ,i use this commands: 105 docker stop $(docker ps -a ) 106 docker stop $(docker ps -aq ) 107 docker system --help 108 docker system prune -f 109 docker volume rm $(docker volume ls -q) 110 docker image rm $(docker image ls -q) 111 rm -rf /etc/ceph /etc/cni /etc/kubernetes /opt/cni /opt/rke
/run/secrets/kubernetes.io /run/calico /run/flannel /var/lib/calico
/var/lib/etcd /var/lib/cni /var/lib/kubelet /var/lib/rancher/rke/log
/var/log/containers /var/log/pods /var/run/calico 112 reboot

then finished i login my user "rke" and rke up，it just same problem, how sholud i need to clean and reinstall ?

superseb commented 3 years ago

rke remove will clean everything as well, did you clean up local files created by RKE as well? (cluster.rkestate and kube_config_cluster.yml) ?

JasinlikeEatingOrange commented 3 years ago

rke remove will clean everything as well, did you clean up local files created by RKE as well? (cluster.rkestate and kube_config_cluster.yml) ?

i did not clean these two files . let me have a try. thx a lot!!

JasinlikeEatingOrange commented 3 years ago

rke remove use rke remove it will auto del cluster.rkestate and kube_config_cluster.yml. but it throw same prblems. host [10.109.31.135] failed to check etcd health: failed to get /health for host [10.109.31.135]: Get "https://10.109.31.135:2379/health": net/http: TLS handshake timeout

superseb commented 3 years ago

Please provide the full log of rke --debug up. Does it work with a normal release (non rc like v1.2.9)?

JasinlikeEatingOrange commented 3 years ago

Please provide the full log of rke --debug up. Does it work with a normal release (non rc like v1.2.9)?

2021-07-14 10:00:42.236397 I | embed: rejected connection from "10.109.31.113:51426" (error "remote error: tls: bad certificate", ServerName "") there is a new error in my docker logs

JasinlikeEatingOrange commented 3 years ago

time="2021-07-14T18:16:48+08:00" level=debug msg="Loglevel set to [debug]"
time="2021-07-14T18:16:48+08:00" level=warning msg="This is not an officially supported version (v1.3.0-rc8) of RKE. Please download the latest official release at https://github.com/rancher/rke/releases"
time="2021-07-14T18:16:48+08:00" level=info msg="Running RKE version: v1.3.0-rc8"
time="2021-07-14T18:16:48+08:00" level=debug msg="audit log policy found in cluster.yml"
time="2021-07-14T18:16:48+08:00" level=info msg="Initiating Kubernetes cluster"
time="2021-07-14T18:16:48+08:00" level=debug msg="Loading data.json from local source"
time="2021-07-14T18:16:48+08:00" level=debug msg="data.json SHA256 checksum: 705b1e1058c996d5bab9ec46a6b8066fe357a22978999a44faffe1e70638c5e3"
time="2021-07-14T18:16:48+08:00" level=debug msg="No DNS provider configured, setting default based on cluster version [1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="DNS provider set to [coredns]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Enabling kube-api audit log for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="No input provided for maxUnavailableWorker, setting it to default value of 10 percent"
time="2021-07-14T18:16:48+08:00" level=debug msg="No input provided for maxUnavailableControlplane, setting it to default value of 1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking ingress default backend for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="[state] previous state found, this is not a legacy cluster"
time="2021-07-14T18:16:48+08:00" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
time="2021-07-14T18:16:48+08:00" level=info msg="[certificates] Generating admin certificates and kubeconfig"
time="2021-07-14T18:16:48+08:00" level=info msg="Successfully Deployed state file at [./cluster.rkestate]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Cluster version [1.20.7-rancher1-1] needs to have kube-api audit log enabled"
time="2021-07-14T18:16:48+08:00" level=debug msg="Enabling kube-api audit log for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.135 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.136 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: controlplane"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: worker"
time="2021-07-14T18:16:48+08:00" level=debug msg="Host: 10.109.31.137 has role: etcd"
time="2021-07-14T18:16:48+08:00" level=info msg="Building Kubernetes cluster"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=info msg="[dialer] Setup tunnel for host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Connecting to Docker API for host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.135]: types.Info{ID:\"7PZW:ENLM:7ZC2:KKWZ:CNEI:GCKI:EHGM:NP2N:HWNQ:ANEQ:2O2W:DTGP\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.417684777+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc0003a83f0), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyprk1\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.137]: types.Info{ID:\"T4WJ:2Y77:ZWUX:BWC5:DQDP:DPBW:ZO2X:U24U:6E2Q:PS2D:KZT5:AH4O\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.466766633+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc000246070), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyp2\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=debug msg="Docker Info found for host [10.109.31.136]: types.Info{ID:\"5Y53:KY6C:RART:JOV2:USAO:YIPT:HBME:PJGF:67O3:36YC:55M5:5JM7\", Containers:3, ContainersRunning:2, ContainersPaused:0, ContainersStopped:1, Images:3, Driver:\"overlay2\", DriverStatus:[][2]string{[2]string{\"Backing Filesystem\", \"xfs\"}, [2]string{\"Supports d_type\", \"true\"}, [2]string{\"Native Overlay Diff\", \"true\"}, [2]string{\"userxattr\", \"false\"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string{\"local\"}, Network:[]string{\"bridge\", \"host\", \"ipvlan\", \"macvlan\", \"null\", \"overlay\"}, Authorization:[]string(nil), Log:[]string{\"awslogs\", \"fluentd\", \"gcplogs\", \"gelf\", \"journald\", \"json-file\", \"local\", \"logentries\", \"splunk\", \"syslog\"}}, MemoryLimit:true, SwapLimit:true, KernelMemory:true, KernelMemoryTCP:true, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:true, CPUSet:true, PidsLimit:true, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:35, OomKillDisable:true, NGoroutines:42, SystemTime:\"2021-07-14T18:16:48.473340071+08:00\", LoggingDriver:\"json-file\", CgroupDriver:\"cgroupfs\", CgroupVersion:\"1\", NEventsListener:0, KernelVersion:\"3.10.0-1160.el7.x86_64\", OperatingSystem:\"CentOS Linux 7 (Core)\", OSVersion:\"7\", OSType:\"linux\", Architecture:\"x86_64\", IndexServerAddress:\"https://index.docker.io/v1/\", RegistryConfig:(*registry.ServiceConfig)(0xc000246230), NCPU:8, MemTotal:8369065984, GenericResources:[]swarm.GenericResource(nil), DockerRootDir:\"/var/lib/docker\", HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\", Name:\"hyp1\", Labels:[]string{}, ExperimentalBuild:false, ServerVersion:\"20.10.7\", ClusterStore:\"\", ClusterAdvertise:\"\", Runtimes:map[string]types.Runtime{\"io.containerd.runc.v2\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"io.containerd.runtime.v1.linux\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}, \"runc\":types.Runtime{Path:\"runc\", Args:[]string(nil), Shim:(*types.ShimConfig)(nil)}}, DefaultRuntime:\"runc\", Swarm:swarm.Info{NodeID:\"\", NodeAddr:\"\", LocalNodeState:\"inactive\", ControlAvailable:false, Error:\"\", RemoteManagers:[]swarm.Peer(nil), Nodes:0, Managers:0, Cluster:(*swarm.ClusterInfo)(nil), Warnings:[]string(nil)}, LiveRestoreEnabled:false, Isolation:\"\", InitBinary:\"docker-init\", ContainerdCommit:types.Commit{ID:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\", Expected:\"d71fcd7d8303cbf684402823e425e9dd2e99285d\"}, RuncCommit:types.Commit{ID:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\", Expected:\"b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7\"}, InitCommit:types.Commit{ID:\"de40ad0\", Expected:\"de40ad0\"}, SecurityOptions:[]string{\"name=seccomp,profile=default\"}, ProductLicense:\"\", DefaultAddressPools:[]types.NetworkAddressPool(nil), Warnings:[]string(nil)}"
time="2021-07-14T18:16:48+08:00" level=info msg="[network] Deploying port listener containers"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:48+08:00" level=debug msg="[network] Starting deployListener [rke-etcd-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:48+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:48+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Service is already up on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=debug msg="[network] Starting deployListener [rke-cp-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:49+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:49+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:50+08:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.109.31.137]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=debug msg="[network] Starting deployListener [rke-worker-port-listener] on host [10.109.31.137]"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:51+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:51+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:52+08:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Port listener containers deployed successfully"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Running etcd <-> etcd port checks"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379 2380], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:53+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:53+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:53+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:54+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:54+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:55+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:55+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Running control plane -> etcd port checks"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [2379], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:55+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:55+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:56+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:56+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Running control plane -> worker port checks"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [10250], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:56+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:56+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:57+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:58+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:58+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Running workers -> control plane port checks"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.136] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.137] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=info msg="[network] Checking if host [10.109.31.135] can connect to host(s) [10.109.31.135 10.109.31.136 10.109.31.137] on port(s) [6443], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:16:58+08:00" level=debug msg="[remove/rke-port-checker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:16:58+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:16:58+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="Starting container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.136], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] containerLog [] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-port-checker] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Length of containerLog is [0] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Checking KubeAPI port Control Plane hosts"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.135"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.136"
time="2021-07-14T18:16:59+08:00" level=debug msg="[network] Checking KubeAPI port [6443] on host: 10.109.31.137"
time="2021-07-14T18:16:59+08:00" level=info msg="[network] Removing port listener containers"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:16:59+08:00" level=debug msg="[remove/rke-etcd-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:16:59+08:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-cp-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="[remove/rke-worker-port-listener] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:00+08:00" level=info msg="[network] Port listener containers removed successfully"
time="2021-07-14T18:17:00+08:00" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:00+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:00+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:02+08:00" level=info msg="Starting container [cert-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:02+08:00" level=debug msg="[certificates] Successfully started Certificate deployer container: cert-deployer"
time="2021-07-14T18:17:02+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.136], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Checking if container [cert-deployer] is running on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Removing container [cert-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.135] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.136] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=debug msg="[reconcile] Rebuilding and updating local kube config, creating new kubeconfig"
time="2021-07-14T18:17:07+08:00" level=debug msg="Deploying admin Kubeconfig locally at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Using ./kube_config_cluster.yml to connect to Kubernetes cluster.."
time="2021-07-14T18:17:07+08:00" level=debug msg="[version] Getting Kubernetes server version.."
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] host [10.109.31.137] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2021-07-14T18:17:07+08:00" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
time="2021-07-14T18:17:07+08:00" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
time="2021-07-14T18:17:07+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:07+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:07+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:08+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:08+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:09+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:09+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.135]"
time="2021-07-14T18:17:09+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:09+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:09+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:10+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:11+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:11+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:12+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:12+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.136]"
time="2021-07-14T18:17:12+08:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="[remove/file-deployer] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:12+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:12+08:00" level=info msg="Starting container [file-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:13+08:00" level=info msg="Successfully started [file-deployer] container on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:13+08:00" level=info msg="Container [file-deployer] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:14+08:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Exit code for [file-deployer] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[remove/file-deployer] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[remove/file-deployer] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="Removing container [file-deployer] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=debug msg="[file-deploy] Successfully deployed file [/etc/kubernetes/audit-policy.yaml] on node [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
time="2021-07-14T18:17:14+08:00" level=info msg="[reconcile] Reconciling cluster state"
time="2021-07-14T18:17:14+08:00" level=info msg="[reconcile] This is newly generated cluster"
time="2021-07-14T18:17:14+08:00" level=debug msg="Encryption is disabled in both current and new spec; no action is required"
time="2021-07-14T18:17:14+08:00" level=info msg="Pre-pulling kubernetes images"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.135]"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.137]"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/hyperkube:v1.20.7-rancher1] exists on host [10.109.31.136]"
time="2021-07-14T18:17:14+08:00" level=info msg="Kubernetes images pulled successfully"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: getting serviceOptions for cluster version [v1.20.7-rancher1-1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v1.20.7-rancher1] from image [rancher/hyperkube:v1.20.7-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="getDefaultKubernetesServicesOptions: serviceOptions found for cluster major version [v1.20]"
time="2021-07-14T18:17:14+08:00" level=debug msg="Extracted version [v3.4.15-rancher1] from image [rancher/mirrored-coreos-etcd:v3.4.15-rancher1]"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.3-rancher99] for advertising port 4001, not going to advertise port 4001"
time="2021-07-14T18:17:14+08:00" level=debug msg="etcd version [3.4.15-rancher1] is higher than max version [3.4.14-rancher99] for adding stricter TLS cipher suites, going to add stricter TLS cipher suites arguments to etcd"
time="2021-07-14T18:17:14+08:00" level=debug msg="Version [3.4.15-rancher1] is equal or higher than version [3.2.99]"
time="2021-07-14T18:17:14+08:00" level=info msg="[etcd] Building up etcd plane.."
time="2021-07-14T18:17:14+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:14+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:15+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:15+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:16+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:16+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:16+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:16+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.135:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:16+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:17+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:17+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:17+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:17+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:18+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.135]"
time="2021-07-14T18:17:23+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:23+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:23+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:24+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.135]"
time="2021-07-14T18:17:24+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:24+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.135]: stderr: [], stdout: []"
time="2021-07-14T18:17:25+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.135] is [0]"
time="2021-07-14T18:17:25+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:25+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:25+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:25+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.135]"
time="2021-07-14T18:17:26+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135], try #1"
time="2021-07-14T18:17:26+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.135], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.135]"
time="2021-07-14T18:17:27+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:27+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:28+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:29+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:29+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:30+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:30+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.136:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:30+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:30+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:31+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:31+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.136]"
time="2021-07-14T18:17:36+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:36+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:37+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:37+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.136]: stderr: [], stdout: []"
time="2021-07-14T18:17:38+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.136] is [0]"
time="2021-07-14T18:17:38+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:38+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:38+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:38+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:39+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:39+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:40+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136], try #1"
time="2021-07-14T18:17:40+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.136]"
time="2021-07-14T18:17:40+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.136], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.136]"
time="2021-07-14T18:17:41+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:41+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:42+08:00" level=info msg="Starting container [etcd-fix-perm] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:43+08:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:43+08:00" level=info msg="Container [etcd-fix-perm] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:44+08:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Exit code for [etcd-fix-perm] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-fix-perm] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-fix-perm] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=info msg="Removing container [etcd-fix-perm] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Container [etcd] is already running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Checking if container [etcd] is eligible for upgrade on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Container [etcd] is not eligible for upgrade on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Extracted version [v0.1.76] from image [rancher/rke-tools:v0.1.76]"
time="2021-07-14T18:17:44+08:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[etcd] Using command [/opt/rke-tools/rke-etcd-backup etcd-backup save --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-node.pem --key /etc/kubernetes/ssl/kube-node-key.pem --name etcd-rolling-snapshots --endpoints=10.109.31.137:2379 --retention=72h --creation=12h] for rolling snapshot container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="[remove/etcd-rolling-snapshots] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=info msg="Removing container [etcd-rolling-snapshots] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:44+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:44+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:45+08:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:45+08:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.109.31.137]"
time="2021-07-14T18:17:50+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:50+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Starting container [rke-bundle-cert] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:51+08:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:51+08:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.109.31.137]: stderr: [], stdout: []"
time="2021-07-14T18:17:52+08:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="Exit code for [rke-bundle-cert] container on host [10.109.31.137] is [0]"
time="2021-07-14T18:17:52+08:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=info msg="Removing container [rke-bundle-cert] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:52+08:00" level=debug msg="[etcd] Creating log link for Container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:52+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:52+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:53+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd-rolling-snapshots] on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[etcd] Creating log link for Container [etcd] on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="[remove/rke-log-linker] Container doesn't exist on host [10.109.31.137]"
time="2021-07-14T18:17:54+08:00" level=debug msg="Checking if image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137], try #1"
time="2021-07-14T18:17:54+08:00" level=info msg="Image [rancher/rke-tools:v0.1.76] exists on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="Starting container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:55+08:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[remove/rke-log-linker] Checking if container is running on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[remove/rke-log-linker] Removing container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="Removing container [rke-log-linker] on host [10.109.31.137], try #1"
time="2021-07-14T18:17:55+08:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=debug msg="[etcd] Successfully created log link for Container [etcd] on host [10.109.31.137]"
time="2021-07-14T18:17:55+08:00" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
time="2021-07-14T18:17:55+08:00" level=debug msg="[etcd] check etcd cluster health on host [10.109.31.135]"
time="2021-07-14T18:18:05+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:21+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:36+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:18:51+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:06+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:21+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:36+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:19:51+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"
time="2021-07-14T18:20:06+08:00" level=debug msg="[etcd] failed to check health for etcd host [10.109.31.135]: failed to get /health for host [10.109.31.135]: Get \"https://10.109.31.135:2379/health\": net/http: TLS handshake timeout"

superseb commented 3 years ago

Please provide the log after running rke remove, this line indicates you still have local state and this is not a fresh/new run:

time="2021-07-14T18:16:48+08:00" level=debug msg="[state] previous state found, this is not a legacy cluster"

stale[bot] commented 2 years ago

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

rancher / rke

[etcd] host [10.109.31.135] failed to check etcd health: failed to get /health for host [10.109.31.135]: Get "https://10.109.31.135:2379/health": net/http: TLS handshake timeout #2606

If you intended to deploy Kubernetes in an air-gapped environment,

please consult the documentation on how to configure custom RKE images.