search

rancher / terraform-provider-rke

Terraform provider plugin for deploy kubernetes cluster by RKE(Rancher Kubernetes Engine)
Mozilla Public License 2.0
340 stars 152 forks source link

Deploying rke cluster fails on provider version 1.3.2 #355

Open Aransh opened 2 years ago

Aransh commented 2 years ago

I am unable to deploy an rke cluster using version 1.3.2 of the provider (with either k8s v1.23.6-rancher1-1 or v1.21.7-rancher1-1). Downgrading to provider version 1.3.1 works (but of course, means k8s 1.23 is not supported).

The main error I'm receiving is 'Failed running cluster err:[ "master-ip-address" not found]' Again, the same exact configuration works just fine with provider version 1.3.1, so I doubt it's an infrastructure issue. Also, I've been able to reproduce the issue on version 1.3.2 10 out of 10 times, while on 1.3.1 (and older versions) it worked every time for months.

Commenting the full error log (as it's longer than github's limit)

Aransh commented 2 years ago 
Error:
============= RKE outputs ==============
time="2022-07-31T12:32:05+03:00" level=info msg="[rke_provider] rke cluster changed arguments: map[addons:true authentication:true bastion_host:true cloud_provider:true cluster_name:true ingress:true kubernetes_version:true network:true nodes:true restore:true services:true ssh_key_path:true]"
time="2022-07-31T12:32:05+03:00" level=info msg="Creating RKE cluster..."
time="2022-07-31T12:32:05+03:00" level=info msg="Initiating Kubernetes cluster"
time="2022-07-31T12:32:05+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.2.233]"
time="2022-07-31T12:32:05+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.6.29]"
time="2022-07-31T12:32:05+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.4.102]"
time="2022-07-31T12:32:08+03:00" level=info msg="Checking if container [cluster-state-deployer] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:32:08+03:00" level=info msg="Pulling image [rancher/rke-tools:v0.1.80] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:16+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:19+03:00" level=info msg="Starting container [cluster-state-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:19+03:00" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [10.145.2.233]"
time="2022-07-31T12:32:20+03:00" level=info msg="Checking if container [cluster-state-deployer] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:32:20+03:00" level=info msg="Pulling image [rancher/rke-tools:v0.1.80] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:28+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:30+03:00" level=info msg="Starting container [cluster-state-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:30+03:00" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [10.145.4.102]"
time="2022-07-31T12:32:30+03:00" level=info msg="Checking if container [cluster-state-deployer] is running on host [10.145.6.29], try #1"
time="2022-07-31T12:32:31+03:00" level=info msg="Pulling image [rancher/rke-tools:v0.1.80] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:39+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:41+03:00" level=info msg="Starting container [cluster-state-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:41+03:00" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [10.145.6.29]"
time="2022-07-31T12:32:41+03:00" level=info msg="[certificates] Generating CA kubernetes certificates"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] Generating Kubernetes API server certificates"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] Generating Service account token key"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] Generating Kube Controller certificates"
time="2022-07-31T12:32:42+03:00" level=info msg="[certificates] Generating Kube Scheduler certificates"
time="2022-07-31T12:32:43+03:00" level=info msg="[certificates] Generating Kube Proxy certificates"
time="2022-07-31T12:32:43+03:00" level=info msg="[certificates] Generating Node certificate"
time="2022-07-31T12:32:43+03:00" level=info msg="[certificates] Generating admin certificates and kubeconfig"
time="2022-07-31T12:32:44+03:00" level=info msg="[certificates] Generating Kubernetes API server proxy client certificates"
time="2022-07-31T12:32:44+03:00" level=info msg="[certificates] Generating kube-etcd-10-145-2-233 certificate and key"
time="2022-07-31T12:32:44+03:00" level=info msg="[certificates] Generating kube-etcd-10-145-4-102 certificate and key"
time="2022-07-31T12:32:44+03:00" level=info msg="[certificates] Generating kube-etcd-10-145-6-29 certificate and key"
time="2022-07-31T12:32:44+03:00" level=info msg="Successfully Deployed state file at [/Users/aransh/Downloads/igentify/terraform-modules/regression_tests/kubernetes_cluster/terraform-provider-rke-tmp-280480591/cluster.rkestate]"
time="2022-07-31T12:32:44+03:00" level=info msg="Building Kubernetes cluster"
time="2022-07-31T12:32:44+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.2.233]"
time="2022-07-31T12:32:44+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.4.102]"
time="2022-07-31T12:32:44+03:00" level=info msg="[dialer] Setup tunnel for host [10.145.6.29]"
time="2022-07-31T12:32:47+03:00" level=info msg="[network] Deploying port listener containers"
time="2022-07-31T12:32:47+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:47+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:47+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:48+03:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:48+03:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:48+03:00" level=info msg="Starting container [rke-etcd-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:48+03:00" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [10.145.2.233]"
time="2022-07-31T12:32:48+03:00" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [10.145.4.102]"
time="2022-07-31T12:32:49+03:00" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [10.145.6.29]"
time="2022-07-31T12:32:49+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:49+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:49+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:49+03:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:49+03:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:49+03:00" level=info msg="Starting container [rke-cp-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:50+03:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.145.2.233]"
time="2022-07-31T12:32:50+03:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.145.4.102]"
time="2022-07-31T12:32:50+03:00" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [10.145.6.29]"
time="2022-07-31T12:32:50+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:50+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:50+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:51+03:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:51+03:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:51+03:00" level=info msg="Starting container [rke-worker-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.145.2.233]"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.145.6.29]"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [10.145.4.102]"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Port listener containers deployed successfully"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Running etcd <-> etcd port checks"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Checking if host [10.145.2.233] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379 2380], try #1"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Checking if host [10.145.4.102] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379 2380], try #1"
time="2022-07-31T12:32:51+03:00" level=info msg="[network] Checking if host [10.145.6.29] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379 2380], try #1"
time="2022-07-31T12:32:52+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:52+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:52+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:52+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:52+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:52+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:52+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.2.233]"
time="2022-07-31T12:32:52+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.4.102]"
time="2022-07-31T12:32:52+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.6.29]"
time="2022-07-31T12:32:53+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="[network] Running control plane -> etcd port checks"
time="2022-07-31T12:32:53+03:00" level=info msg="[network] Checking if host [10.145.2.233] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="[network] Checking if host [10.145.6.29] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="[network] Checking if host [10.145.4.102] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [2379], try #1"
time="2022-07-31T12:32:53+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:53+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:53+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:54+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.2.233]"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.4.102]"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.6.29]"
time="2022-07-31T12:32:54+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Running control plane -> worker port checks"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Checking if host [10.145.2.233] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [10250], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Checking if host [10.145.4.102] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [10250], try #1"
time="2022-07-31T12:32:54+03:00" level=info msg="[network] Checking if host [10.145.6.29] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [10250], try #1"
time="2022-07-31T12:32:55+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:55+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:55+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:55+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:55+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:55+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.2.233]"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.4.102]"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.6.29]"
time="2022-07-31T12:32:56+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Running workers -> control plane port checks"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Checking if host [10.145.4.102] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [6443], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Checking if host [10.145.2.233] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [6443], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="[network] Checking if host [10.145.6.29] can connect to host(s) [10.145.2.233 10.145.4.102 10.145.6.29] on port(s) [6443], try #1"
time="2022-07-31T12:32:56+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:32:56+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:32:56+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:32:57+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:57+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:57+03:00" level=info msg="Starting container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:57+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.2.233]"
time="2022-07-31T12:32:57+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.4.102]"
time="2022-07-31T12:32:57+03:00" level=info msg="[network] Successfully started [rke-port-checker] container on host [10.145.6.29]"
time="2022-07-31T12:32:57+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:57+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:57+03:00" level=info msg="Removing container [rke-port-checker] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="[network] Skipping kubeapi port check"
time="2022-07-31T12:32:58+03:00" level=info msg="[network] Removing port listener containers"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-etcd-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:32:58+03:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:32:58+03:00" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:58+03:00" level=info msg="Removing container [rke-cp-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:32:59+03:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.145.2.233], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.145.4.102], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="Removing container [rke-worker-port-listener] on host [10.145.6.29], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:32:59+03:00" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:32:59+03:00" level=info msg="[network] Port listener containers removed successfully"
time="2022-07-31T12:32:59+03:00" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
time="2022-07-31T12:32:59+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.6.29], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:32:59+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:33:00+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:33:00+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:33:00+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:33:00+03:00" level=info msg="Starting container [cert-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:00+03:00" level=info msg="Starting container [cert-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:00+03:00" level=info msg="Starting container [cert-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:00+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:33:01+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:33:01+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.6.29], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="Checking if container [cert-deployer] is running on host [10.145.6.29], try #1"
Aransh commented 2 years ago 
time="2022-07-31T12:33:06+03:00" level=info msg="Removing container [cert-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="Removing container [cert-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="Removing container [cert-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:06+03:00" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2022-07-31T12:33:06+03:00" level=info msg="Successfully Deployed local admin kubeconfig at [/Users/aransh/Downloads/igentify/terraform-modules/regression_tests/kubernetes_cluster/terraform-provider-rke-tmp-280480591/kube_config_cluster.yml]"
time="2022-07-31T12:33:07+03:00" level=warning msg="[reconcile] host [10.145.2.233] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2022-07-31T12:33:07+03:00" level=info msg="Successfully Deployed local admin kubeconfig at [/Users/aransh/Downloads/igentify/terraform-modules/regression_tests/kubernetes_cluster/terraform-provider-rke-tmp-280480591/kube_config_cluster.yml]"
time="2022-07-31T12:33:08+03:00" level=warning msg="[reconcile] host [10.145.4.102] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2022-07-31T12:33:08+03:00" level=info msg="Successfully Deployed local admin kubeconfig at [/Users/aransh/Downloads/igentify/terraform-modules/regression_tests/kubernetes_cluster/terraform-provider-rke-tmp-280480591/kube_config_cluster.yml]"
time="2022-07-31T12:33:10+03:00" level=warning msg="[reconcile] host [10.145.6.29] is a control plane node without reachable Kubernetes API endpoint in the cluster"
time="2022-07-31T12:33:10+03:00" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
time="2022-07-31T12:33:10+03:00" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
time="2022-07-31T12:33:10+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/cloud-config] to node [10.145.2.233]"
time="2022-07-31T12:33:10+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:33:10+03:00" level=info msg="Starting container [file-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:11+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.2.233]"
time="2022-07-31T12:33:11+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.2.233]"
time="2022-07-31T12:33:11+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.2.233]"
time="2022-07-31T12:33:11+03:00" level=info msg="Removing container [file-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:11+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:33:11+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/cloud-config] to node [10.145.4.102]"
time="2022-07-31T12:33:12+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:33:12+03:00" level=info msg="Starting container [file-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:13+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.4.102]"
time="2022-07-31T12:33:13+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.4.102]"
time="2022-07-31T12:33:13+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.4.102]"
time="2022-07-31T12:33:13+03:00" level=info msg="Removing container [file-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:13+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:33:13+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/cloud-config] to node [10.145.6.29]"
time="2022-07-31T12:33:14+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:33:14+03:00" level=info msg="Starting container [file-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:15+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.6.29]"
time="2022-07-31T12:33:15+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.6.29]"
time="2022-07-31T12:33:15+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.6.29]"
time="2022-07-31T12:33:15+03:00" level=info msg="Removing container [file-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:15+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:33:15+03:00" level=info msg="[/etc/kubernetes/cloud-config] Successfully deployed kubernetes cloud config to Cluster nodes"
time="2022-07-31T12:33:15+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.145.2.233]"
time="2022-07-31T12:33:16+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:33:16+03:00" level=info msg="Starting container [file-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:17+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.2.233]"
time="2022-07-31T12:33:17+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.2.233]"
time="2022-07-31T12:33:17+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.2.233]"
time="2022-07-31T12:33:18+03:00" level=info msg="Removing container [file-deployer] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:18+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:33:18+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.145.4.102]"
time="2022-07-31T12:33:18+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:33:18+03:00" level=info msg="Starting container [file-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:19+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.4.102]"
time="2022-07-31T12:33:19+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.4.102]"
time="2022-07-31T12:33:19+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.4.102]"
time="2022-07-31T12:33:19+03:00" level=info msg="Removing container [file-deployer] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:19+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:33:19+03:00" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [10.145.6.29]"
time="2022-07-31T12:33:20+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:33:20+03:00" level=info msg="Starting container [file-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:21+03:00" level=info msg="Successfully started [file-deployer] container on host [10.145.6.29]"
time="2022-07-31T12:33:21+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.6.29]"
time="2022-07-31T12:33:21+03:00" level=info msg="Waiting for [file-deployer] container to exit on host [10.145.6.29]"
time="2022-07-31T12:33:22+03:00" level=info msg="Removing container [file-deployer] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:22+03:00" level=info msg="[remove/file-deployer] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:33:22+03:00" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
time="2022-07-31T12:33:22+03:00" level=info msg="[reconcile] Reconciling cluster state"
time="2022-07-31T12:33:22+03:00" level=info msg="[reconcile] This is newly generated cluster"
time="2022-07-31T12:33:22+03:00" level=info msg="Pre-pulling kubernetes images"
time="2022-07-31T12:33:22+03:00" level=info msg="Pulling image [rancher/hyperkube:v1.23.6-rancher1] on host [10.145.2.233], try #1"
time="2022-07-31T12:33:22+03:00" level=info msg="Pulling image [rancher/hyperkube:v1.23.6-rancher1] on host [10.145.4.102], try #1"
time="2022-07-31T12:33:22+03:00" level=info msg="Pulling image [rancher/hyperkube:v1.23.6-rancher1] on host [10.145.6.29], try #1"
time="2022-07-31T12:33:57+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:33:58+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:33:59+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:33:59+03:00" level=info msg="Kubernetes images pulled successfully"
time="2022-07-31T12:33:59+03:00" level=info msg="[etcd] Building up etcd plane.."
time="2022-07-31T12:33:59+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:34:04+03:00" level=info msg="Starting container [etcd-fix-perm] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:05+03:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.145.2.233]"
time="2022-07-31T12:34:05+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.2.233]"
time="2022-07-31T12:34:05+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.2.233]"
time="2022-07-31T12:34:05+03:00" level=info msg="Removing container [etcd-fix-perm] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:05+03:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:34:05+03:00" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:12+03:00" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.145.2.233]"
time="2022-07-31T12:34:13+03:00" level=info msg="Starting container [etcd] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:14+03:00" level=info msg="[etcd] Successfully started [etcd] container on host [10.145.2.233]"
time="2022-07-31T12:34:14+03:00" level=info msg="[etcd] Snapshots configured to S3 compatible backend at [s3.amazonaws.com] to bucket [igentify-test-kubernetes-etcd-backup] using accesskey [AKIAXRCQ6HXIS4PBSLQY] and using region [eu-west-1]"
time="2022-07-31T12:34:14+03:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.145.2.233]"
time="2022-07-31T12:34:14+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:34:14+03:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:15+03:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.145.2.233]"
time="2022-07-31T12:34:20+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:34:21+03:00" level=info msg="Starting container [rke-bundle-cert] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:21+03:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.145.2.233]"
time="2022-07-31T12:34:21+03:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.145.2.233]"
time="2022-07-31T12:34:21+03:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.145.2.233]"
time="2022-07-31T12:34:21+03:00" level=info msg="Removing container [rke-bundle-cert] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:22+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:34:22+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:23+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:34:23+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:23+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:34:23+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:34:24+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:24+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:34:24+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:34:25+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:34:25+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:34:28+03:00" level=info msg="Starting container [etcd-fix-perm] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:28+03:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.145.4.102]"
time="2022-07-31T12:34:28+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.4.102]"
time="2022-07-31T12:34:28+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.4.102]"
time="2022-07-31T12:34:29+03:00" level=info msg="Removing container [etcd-fix-perm] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:29+03:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:34:29+03:00" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:35+03:00" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.145.4.102]"
time="2022-07-31T12:34:36+03:00" level=info msg="Starting container [etcd] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:37+03:00" level=info msg="[etcd] Successfully started [etcd] container on host [10.145.4.102]"
time="2022-07-31T12:34:37+03:00" level=info msg="[etcd] Snapshots configured to S3 compatible backend at [s3.amazonaws.com] to bucket [igentify-test-kubernetes-etcd-backup] using accesskey [AKIAXRCQ6HXIS4PBSLQY] and using region [eu-west-1]"
time="2022-07-31T12:34:37+03:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.145.4.102]"
time="2022-07-31T12:34:37+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:34:37+03:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:38+03:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.145.4.102]"
time="2022-07-31T12:34:43+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:34:44+03:00" level=info msg="Starting container [rke-bundle-cert] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:44+03:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.145.4.102]"
time="2022-07-31T12:34:44+03:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.145.4.102]"
time="2022-07-31T12:34:44+03:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.145.4.102]"
time="2022-07-31T12:34:44+03:00" level=info msg="Removing container [rke-bundle-cert] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:45+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:34:45+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:46+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:34:46+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:46+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:34:47+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:34:47+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:48+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:34:48+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:34:48+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:34:51+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:34:51+03:00" level=info msg="Starting container [etcd-fix-perm] on host [10.145.6.29], try #1"
time="2022-07-31T12:34:52+03:00" level=info msg="Successfully started [etcd-fix-perm] container on host [10.145.6.29]"
time="2022-07-31T12:34:52+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.6.29]"
time="2022-07-31T12:34:52+03:00" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [10.145.6.29]"
time="2022-07-31T12:34:52+03:00" level=info msg="Removing container [etcd-fix-perm] on host [10.145.6.29], try #1"
time="2022-07-31T12:34:52+03:00" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:34:52+03:00" level=info msg="Pulling image [rancher/mirrored-coreos-etcd:v3.5.3] on host [10.145.6.29], try #1"
time="2022-07-31T12:34:59+03:00" level=info msg="Image [rancher/mirrored-coreos-etcd:v3.5.3] exists on host [10.145.6.29]"
time="2022-07-31T12:35:00+03:00" level=info msg="Starting container [etcd] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:00+03:00" level=info msg="[etcd] Successfully started [etcd] container on host [10.145.6.29]"
time="2022-07-31T12:35:00+03:00" level=info msg="[etcd] Snapshots configured to S3 compatible backend at [s3.amazonaws.com] to bucket [igentify-test-kubernetes-etcd-backup] using accesskey [AKIAXRCQ6HXIS4PBSLQY] and using region [eu-west-1]"
time="2022-07-31T12:35:00+03:00" level=info msg="[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [10.145.6.29]"
time="2022-07-31T12:35:01+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:01+03:00" level=info msg="Starting container [etcd-rolling-snapshots] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:02+03:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.145.6.29]"
time="2022-07-31T12:35:07+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:08+03:00" level=info msg="Starting container [rke-bundle-cert] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:08+03:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.145.6.29]"
time="2022-07-31T12:35:08+03:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.145.6.29]"
time="2022-07-31T12:35:08+03:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.145.6.29]"
time="2022-07-31T12:35:08+03:00" level=info msg="Removing container [rke-bundle-cert] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:09+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:09+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:10+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:10+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:10+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:11+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:11+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:11+03:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:12+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:12+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:12+03:00" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
time="2022-07-31T12:35:14+03:00" level=info msg="[etcd] etcd host [10.145.2.233] reported healthy=true"
time="2022-07-31T12:35:14+03:00" level=info msg="[controlplane] Building up Controller Plane.."
time="2022-07-31T12:35:14+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:35:14+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:35:14+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.6.29], try #1"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:35:15+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:35:15+03:00" level=info msg="Starting container [kube-apiserver] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:16+03:00" level=info msg="Starting container [kube-apiserver] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:16+03:00" level=info msg="Starting container [kube-apiserver] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:16+03:00" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [10.145.2.233]"
time="2022-07-31T12:35:16+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.145.2.233]"
time="2022-07-31T12:35:16+03:00" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [10.145.4.102]"
time="2022-07-31T12:35:16+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.145.4.102]"
time="2022-07-31T12:35:16+03:00" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [10.145.6.29]"
time="2022-07-31T12:35:16+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.145.6.29]"
time="2022-07-31T12:35:24+03:00" level=info msg="[healthcheck] service [kube-apiserver] on host [10.145.2.233] is healthy"
time="2022-07-31T12:35:24+03:00" level=info msg="[healthcheck] service [kube-apiserver] on host [10.145.4.102] is healthy"
time="2022-07-31T12:35:24+03:00" level=info msg="[healthcheck] service [kube-apiserver] on host [10.145.6.29] is healthy"
time="2022-07-31T12:35:25+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:25+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:25+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:25+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:25+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:25+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:26+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:35:26+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:35:26+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:26+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:26+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:26+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:35:26+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:26+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:35:26+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:26+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:35:26+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:35:26+03:00" level=info msg="Starting container [kube-controller-manager] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:26+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:35:26+03:00" level=info msg="Starting container [kube-controller-manager] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:27+03:00" level=info msg="Starting container [kube-controller-manager] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:27+03:00" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [10.145.2.233]"
time="2022-07-31T12:35:27+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.145.2.233]"
time="2022-07-31T12:35:27+03:00" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [10.145.4.102]"
time="2022-07-31T12:35:27+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.145.4.102]"
time="2022-07-31T12:35:27+03:00" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [10.145.6.29]"
time="2022-07-31T12:35:27+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.145.6.29]"
time="2022-07-31T12:35:29+03:00" level=info msg="[healthcheck] service [kube-controller-manager] on host [10.145.2.233] is healthy"
time="2022-07-31T12:35:29+03:00" level=info msg="[healthcheck] service [kube-controller-manager] on host [10.145.6.29] is healthy"
time="2022-07-31T12:35:29+03:00" level=info msg="[healthcheck] service [kube-controller-manager] on host [10.145.4.102] is healthy"
time="2022-07-31T12:35:30+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:30+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:30+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:30+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:30+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:30+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:30+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:35:31+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:31+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:35:31+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:31+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:35:31+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:35:31+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:31+03:00" level=info msg="Starting container [kube-scheduler] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:31+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:31+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:31+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:35:31+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:35:32+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:35:32+03:00" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [10.145.2.233]"
time="2022-07-31T12:35:32+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.145.2.233]"
time="2022-07-31T12:35:32+03:00" level=info msg="Starting container [kube-scheduler] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:32+03:00" level=info msg="Starting container [kube-scheduler] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:32+03:00" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [10.145.4.102]"
time="2022-07-31T12:35:32+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.145.4.102]"
time="2022-07-31T12:35:32+03:00" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [10.145.6.29]"
time="2022-07-31T12:35:32+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.145.6.29]"
time="2022-07-31T12:35:34+03:00" level=info msg="[healthcheck] service [kube-scheduler] on host [10.145.2.233] is healthy"
time="2022-07-31T12:35:35+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:35+03:00" level=info msg="[healthcheck] service [kube-scheduler] on host [10.145.4.102] is healthy"
time="2022-07-31T12:35:35+03:00" level=info msg="[healthcheck] service [kube-scheduler] on host [10.145.6.29] is healthy"
time="2022-07-31T12:35:35+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:35+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:35+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:36+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:36+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:36+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:35:36+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:36+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:35:36+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:35:36+03:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:36+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:36+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:35:36+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:37+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:37+03:00" level=info msg="[controlplane] Successfully started Controller Plane.."
time="2022-07-31T12:35:37+03:00" level=info msg="[authz] Creating rke-job-deployer ServiceAccount"
time="2022-07-31T12:35:38+03:00" level=info msg="[authz] rke-job-deployer ServiceAccount created successfully"
time="2022-07-31T12:35:38+03:00" level=info msg="[authz] Creating system:node ClusterRoleBinding"
time="2022-07-31T12:35:39+03:00" level=info msg="[authz] system:node ClusterRoleBinding created successfully"
time="2022-07-31T12:35:39+03:00" level=info msg="[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding"
time="2022-07-31T12:35:39+03:00" level=info msg="[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully"
time="2022-07-31T12:35:39+03:00" level=info msg="Successfully Deployed state file at [/Users/aransh/Downloads/igentify/terraform-modules/regression_tests/kubernetes_cluster/terraform-provider-rke-tmp-280480591/cluster.rkestate]"
time="2022-07-31T12:35:39+03:00" level=info msg="[state] Saving full cluster state to Kubernetes"
time="2022-07-31T12:35:41+03:00" level=info msg="[state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state"
time="2022-07-31T12:35:41+03:00" level=info msg="[worker] Building up Worker Plane.."
time="2022-07-31T12:35:41+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.6.29], try #1"
time="2022-07-31T12:35:41+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.4.102], try #1"
time="2022-07-31T12:35:41+03:00" level=info msg="Checking if container [service-sidekick] is running on host [10.145.2.233], try #1"
time="2022-07-31T12:35:41+03:00" level=info msg="[sidekick] Sidekick container already created on host [10.145.2.233]"
time="2022-07-31T12:35:41+03:00" level=info msg="[sidekick] Sidekick container already created on host [10.145.4.102]"
time="2022-07-31T12:35:41+03:00" level=info msg="[sidekick] Sidekick container already created on host [10.145.6.29]"
time="2022-07-31T12:35:41+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:35:41+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:35:41+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:35:41+03:00" level=info msg="Starting container [kubelet] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:41+03:00" level=info msg="Starting container [kubelet] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:42+03:00" level=info msg="Starting container [kubelet] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:42+03:00" level=info msg="[worker] Successfully started [kubelet] container on host [10.145.2.233]"
time="2022-07-31T12:35:42+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [10.145.2.233]"
time="2022-07-31T12:35:42+03:00" level=info msg="[worker] Successfully started [kubelet] container on host [10.145.4.102]"
time="2022-07-31T12:35:42+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [10.145.4.102]"
time="2022-07-31T12:35:42+03:00" level=info msg="[worker] Successfully started [kubelet] container on host [10.145.6.29]"
time="2022-07-31T12:35:42+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [10.145.6.29]"
time="2022-07-31T12:35:44+03:00" level=info msg="[healthcheck] service [kubelet] on host [10.145.4.102] is healthy"
time="2022-07-31T12:35:44+03:00" level=info msg="[healthcheck] service [kubelet] on host [10.145.2.233] is healthy"
time="2022-07-31T12:35:45+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:45+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:45+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:45+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:45+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:35:45+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:35:46+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:46+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:46+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:35:46+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:35:46+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.4.102]"
time="2022-07-31T12:35:46+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.2.233]"
time="2022-07-31T12:35:46+03:00" level=info msg="Starting container [kube-proxy] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:46+03:00" level=info msg="Starting container [kube-proxy] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:46+03:00" level=info msg="[worker] Successfully started [kube-proxy] container on host [10.145.4.102]"
time="2022-07-31T12:35:46+03:00" level=info msg="[worker] Successfully started [kube-proxy] container on host [10.145.2.233]"
time="2022-07-31T12:35:46+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.145.4.102]"
time="2022-07-31T12:35:46+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.145.2.233]"
time="2022-07-31T12:35:49+03:00" level=info msg="[healthcheck] service [kube-proxy] on host [10.145.4.102] is healthy"
time="2022-07-31T12:35:49+03:00" level=info msg="[healthcheck] service [kube-proxy] on host [10.145.2.233] is healthy"
time="2022-07-31T12:35:49+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:49+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:50+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:50+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:50+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.4.102]"
time="2022-07-31T12:35:51+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.2.233]"
time="2022-07-31T12:35:51+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:51+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:51+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:35:51+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:35:51+03:00" level=info msg="[healthcheck] service [kubelet] on host [10.145.6.29] is healthy"
time="2022-07-31T12:35:52+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:52+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:53+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:53+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:53+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:53+03:00" level=info msg="Image [rancher/hyperkube:v1.23.6-rancher1] exists on host [10.145.6.29]"
time="2022-07-31T12:35:54+03:00" level=info msg="Starting container [kube-proxy] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:54+03:00" level=info msg="[worker] Successfully started [kube-proxy] container on host [10.145.6.29]"
time="2022-07-31T12:35:54+03:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.145.6.29]"
time="2022-07-31T12:35:56+03:00" level=info msg="[healthcheck] service [kube-proxy] on host [10.145.6.29] is healthy"
time="2022-07-31T12:35:57+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:57+03:00" level=info msg="Starting container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:58+03:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.145.6.29]"
time="2022-07-31T12:35:58+03:00" level=info msg="Removing container [rke-log-linker] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:58+03:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:35:58+03:00" level=info msg="[worker] Successfully started Worker Plane.."
time="2022-07-31T12:35:58+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.4.102]"
time="2022-07-31T12:35:58+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.2.233]"
time="2022-07-31T12:35:58+03:00" level=info msg="Image [rancher/rke-tools:v0.1.80] exists on host [10.145.6.29]"
time="2022-07-31T12:35:59+03:00" level=info msg="Starting container [rke-log-cleaner] on host [10.145.2.233], try #1"
time="2022-07-31T12:35:59+03:00" level=info msg="Starting container [rke-log-cleaner] on host [10.145.4.102], try #1"
time="2022-07-31T12:35:59+03:00" level=info msg="Starting container [rke-log-cleaner] on host [10.145.6.29], try #1"
time="2022-07-31T12:35:59+03:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.145.2.233]"
time="2022-07-31T12:35:59+03:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.145.4.102]"
time="2022-07-31T12:36:00+03:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.145.6.29]"
time="2022-07-31T12:36:00+03:00" level=info msg="Removing container [rke-log-cleaner] on host [10.145.4.102], try #1"
time="2022-07-31T12:36:00+03:00" level=info msg="Removing container [rke-log-cleaner] on host [10.145.2.233], try #1"
time="2022-07-31T12:36:00+03:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.145.2.233]"
time="2022-07-31T12:36:00+03:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.145.4.102]"
time="2022-07-31T12:36:00+03:00" level=info msg="Removing container [rke-log-cleaner] on host [10.145.6.29], try #1"
time="2022-07-31T12:36:00+03:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.145.6.29]"
time="2022-07-31T12:36:00+03:00" level=info msg="[sync] Syncing nodes Labels and Taints"

Failed running cluster err:[ "10.145.4.102" not found]
========================================

  on ../../kubernetes_cluster/rke_cluster.tf line 26, in resource "rke_cluster" "cluster":
  26: resource "rke_cluster" "cluster" {
Aransh commented 2 years ago

Update - this is still happening with version 1.3.3

richiesebastian commented 2 years ago

facing same issue - do we have a solution yet ?

johnhpatton commented 1 year ago

Same issue here on provider version 1.3.4 using AWS EC2s.

a-blender commented 1 year ago

@Aransh Are you still seeing this issue? Can you please try on the latest version v1.3.5?

Aransh commented 1 year ago

@a-blender posted this over a year ago 😅 No longer working at where this was a problem, maybe @johnhpatton or @richiesebastian can assist?