search

ShubhamTatvamasi / magma-galaxy

https://galaxy.ansible.com/shubhamtatvamasi/magma
BSD 3-Clause "New" or "Revised" License
13 stars 20 forks source link

RKE Issue #3

Closed assyafii closed 2 years ago

assyafii commented 2 years ago

Firstly thankyou for great magma deployment, currently we have an issue when Deploy RKE, you have this issue also or not ?

TASK [kubernetes : Install Rancher Kubernetes Engine] **** fatal: [172.12.12.100]: FAILED! => {"changed": true, "cmd": ["rke", "up", "--config", "/root/magma-galaxy/rke/cluster.yml"], "delta": "0:00:00.163217", "end": "2022-06-17 10:03:59.451131", "msg": "non-zero return code", "rc": 1, "start": "2022-06-17 10:03:59.287914", "stderr": "", "stderr_lines": [], "stdout": "time=\"2022-06-17T10:03:59Z\" level=info msg=\"Running RKE version: v1.3.9\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"Initiating Kubernetes cluster\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"Successfully Deployed state file at [/root/magma-galaxy/rke/cluster.rkestate]\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"Building Kubernetes cluster\"\ntime=\"2022-06-17T10:03:59Z\" level=info msg=\"[dialer] Setup tunnel for host [172.12.12.100]\"\ntime=\"2022-06-17T10:03:59Z\" level=warning msg=\"Failed to set up SSH tunneling for host [172.12.12.100]: Can't retrieve Docker Info: error during connect: Get \\"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info\\": Unable to access node with address [172.12.12.100:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\"\ntime=\"2022-06-17T10:03:59Z\" level=warning msg=\"Removing host [172.12.12.100] from node lists\"\ntime=\"2022-06-17T10:03:59Z\" level=fatal msg=\"Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [172.12.12.100]\"", "stdout_lines": ["time=\"2022-06-17T10:03:59Z\" level=info msg=\"Running RKE version: v1.3.9\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"Initiating Kubernetes cluster\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"Successfully Deployed state file at [/root/magma-galaxy/rke/cluster.rkestate]\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"Building Kubernetes cluster\"", "time=\"2022-06-17T10:03:59Z\" level=info msg=\"[dialer] Setup tunnel for host [172.12.12.100]\"", "time=\"2022-06-17T10:03:59Z\" level=warning msg=\"Failed to set up SSH tunneling for host [172.12.12.100]: Can't retrieve Docker Info: error during connect: Get \\"http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info\\": Unable to access node with address [172.12.12.100:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain\"", "time=\"2022-06-17T10:03:59Z\" level=warning msg=\"Removing host [172.12.12.100] from node lists\"", "time=\"2022-06-17T10:03:59Z\" level=fatal msg=\"Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [172.12.12.100]\""]}

ShubhamTatvamasi commented 2 years ago

Hi @assyafii, can you please try deploying this using Multipass? https://github.com/ShubhamTatvamasi/magma-galaxy/blob/master/docs/multipass.md

haidarx commented 2 years ago

@assyafii I got the same error because in the template, the username used to SSH is "ubuntu". You can try to change the value in magma-galaxy\roles\kubernetes\templates\cluster.yml.j2. The value you should change is

nodes:
    user: ssh-username
Darlanewe commented 7 months ago

hi everyone, sorry for reopening and also if i'm posting in the wrong place but i'm getting an error in RKE step but is different from the above if someone could help

TASK [kubernetes : Install Rancher Kubernetes Engine] **Tuesday 16 January 2024 16:55:53 -0300 (0:00:00.300) 0:02:02.656 *** fatal: [172.17.145.181]: FAILED! => {"changed": true, "cmd": ["rke", "up", "--config", "/home/magma/magma-deployer/rke/cluster.yml"], "delta": "0:02:31.353469", "end": "2024-01-16 16:58:24.921380", "msg": "non-zero return code", "rc": 1, "start": "2024-01-16 16:55:53.567911", "stderr": "", "stderr_lines": [], "stdout": "time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Running RKE version: v1.4.3\"\ntime=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Initiating Kubernetes cluster\"\ntime=\"2024-01-16T16:55:53-03:00\" level=info msg=\"[dialer] Setup tunnel for host [172.17.145.181]\"\ntime=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Finding container [cluster-state-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.88] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:10-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"Starting container [cluster-state-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[state] Successfully started [cluster-state-deployer] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating CA kubernetes certificates\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server certificates\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Service account token key\"\ntime=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kube Controller certificates\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kube Scheduler certificates\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kube Proxy certificates\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Node certificate\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server proxy client certificates\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating kube-etcd-172-17-145-181 certificate and key\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Successfully Deployed state file at [/home/magma/magma-deployer/rke/cluster.rkestate]\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Building Kubernetes cluster\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[dialer] Setup tunnel for host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[network] Deploying port listener containers\"\ntime=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Starting container [rke-etcd-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:13-03:00\" level=info msg=\"[network] Successfully started [rke-etcd-port-listener] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Starting container [rke-cp-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:14-03:00\" level=info msg=\"[network] Successfully started [rke-cp-port-listener] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:14-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:14-03:00\" level=info msg=\"Starting container [rke-worker-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Successfully started [rke-worker-port-listener] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Port listener containers deployed successfully\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Running control plane -> etcd port checks\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [2379], try #1\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Running control plane -> worker port checks\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [10250], try #1\"\ntime=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Running workers -> control plane port checks\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [6443], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Checking KubeAPI port Control Plane hosts\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Removing port listener containers\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-etcd-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[remove/rke-etcd-port-listener] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-cp-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[remove/rke-cp-port-listener] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Removing container [rke-worker-port-listener] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[remove/rke-worker-port-listener] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[network] Port listener containers removed successfully\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:18-03:00\" level=info msg=\"Starting container [cert-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:18-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Removing container [cert-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Successfully Deployed local admin kubeconfig at [/home/magma/magma-deployer/rke/kube_config_cluster.yml]\"\ntime=\"2024-01-16T16:56:23-03:00\" level=warning msg=\"[reconcile] host [172.17.145.181] is a control plane node without reachable Kubernetes API endpoint in the cluster\"\ntime=\"2024-01-16T16:56:23-03:00\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.17.145.181]\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Starting container [file-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Successfully started [file-deployer] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Waiting for [file-deployer] container to exit on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Waiting for [file-deployer] container to exit on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Removing container [file-deployer] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[reconcile] Reconciling cluster state\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[reconcile] This is newly generated cluster\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Pre-pulling kubernetes images\"\ntime=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Pulling image [rancher/hyperkube:v1.24.10-rancher4] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Kubernetes images pulled successfully\"\ntime=\"2024-01-16T16:57:39-03:00\" level=info msg=\"[etcd] Building up etcd plane..\"\ntime=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Starting container [etcd-fix-perm] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Container [etcd-fix-perm] is still running on host [172.17.145.181]: stderr: [], stdout: []\"\ntime=\"2024-01-16T16:57:41-03:00\" level=info msg=\"Removing container [etcd-fix-perm] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:41-03:00\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:41-03:00\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.5.4] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.5.4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Starting container [etcd] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Successfully started [etcd] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:02-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Starting container [rke-bundle-cert] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:03-03:00\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Container [rke-bundle-cert] is still running on host [172.17.145.181]: stderr: [], stdout: []\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Removing container [rke-bundle-cert] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[etcd] etcd host [172.17.145.181] reported healthy=true\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[controlplane] Building up Controller Plane..\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Finding container [service-sidekick] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Starting container [kube-apiserver] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:11-03:00\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [172.17.145.181] is healthy\"\ntime=\"2024-01-16T16:58:11-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:11-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Starting container [kube-controller-manager] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:17-03:00\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [172.17.145.181] is healthy\"\ntime=\"2024-01-16T16:58:17-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:17-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Starting container [kube-scheduler] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:23-03:00\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [172.17.145.181] is healthy\"\ntime=\"2024-01-16T16:58:23-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating rke-job-deployer ServiceAccount\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] rke-job-deployer ServiceAccount created successfully\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating system:node ClusterRoleBinding\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] system:node ClusterRoleBinding created successfully\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Successfully Deployed state file at [/home/magma/magma-deployer/rke/cluster.rkestate]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[state] Saving full cluster state to Kubernetes\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[worker] Building up Worker Plane..\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Finding container [service-sidekick] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[sidekick] Sidekick container already created on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #1\"\ntime=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #2\"\ntime=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"\ntime=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #3\"\ntime=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"\ntime=\"2024-01-16T16:58:24-03:00\" level=fatal msg=\"[workerPlane] Failed to bring up Worker Plane: [Failed to start [kubelet] container on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount]\"", "stdout_lines": ["time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Running RKE version: v1.4.3\"", "time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Initiating Kubernetes cluster\"", "time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"[dialer] Setup tunnel for host [172.17.145.181]\"", "time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Finding container [cluster-state-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:55:53-03:00\" level=info msg=\"Pulling image [rancher/rke-tools:v0.1.88] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:10-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"Starting container [cluster-state-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[state] Successfully started [cluster-state-deployer] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating CA kubernetes certificates\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server certificates\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Service account token key\"", "time=\"2024-01-16T16:56:11-03:00\" level=info msg=\"[certificates] Generating Kube Controller certificates\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kube Scheduler certificates\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kube Proxy certificates\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Node certificate\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating admin certificates and kubeconfig\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating Kubernetes API server proxy client certificates\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[certificates] Generating kube-etcd-172-17-145-181 certificate and key\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Successfully Deployed state file at [/home/magma/magma-deployer/rke/cluster.rkestate]\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Building Kubernetes cluster\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[dialer] Setup tunnel for host [172.17.145.181]\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"[network] Deploying port listener containers\"", "time=\"2024-01-16T16:56:12-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Starting container [rke-etcd-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:13-03:00\" level=info msg=\"[network] Successfully started [rke-etcd-port-listener] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:13-03:00\" level=info msg=\"Starting container [rke-cp-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:14-03:00\" level=info msg=\"[network] Successfully started [rke-cp-port-listener] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:14-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:14-03:00\" level=info msg=\"Starting container [rke-worker-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Successfully started [rke-worker-port-listener] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Port listener containers deployed successfully\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Running control plane -> etcd port checks\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [2379], try #1\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Running control plane -> worker port checks\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [10250], try #1\"", "time=\"2024-01-16T16:56:15-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Running workers -> control plane port checks\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Checking if host [172.17.145.181] can connect to host(s) [172.17.145.181] on port(s) [6443], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Starting container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Successfully started [rke-port-checker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-port-checker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Checking KubeAPI port Control Plane hosts\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[network] Removing port listener containers\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-etcd-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"[remove/rke-etcd-port-listener] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:16-03:00\" level=info msg=\"Removing container [rke-cp-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[remove/rke-cp-port-listener] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Removing container [rke-worker-port-listener] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[remove/rke-worker-port-listener] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[network] Port listener containers removed successfully\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"[certificates] Deploying kubernetes certificates to Cluster nodes\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:17-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:18-03:00\" level=info msg=\"Starting container [cert-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:18-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Finding container [cert-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Removing container [cert-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[reconcile] Rebuilding and updating local kube config\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Successfully Deployed local admin kubeconfig at [/home/magma/magma-deployer/rke/kube_config_cluster.yml]\"", "time=\"2024-01-16T16:56:23-03:00\" level=warning msg=\"[reconcile] host [172.17.145.181] is a control plane node without reachable Kubernetes API endpoint in the cluster\"", "time=\"2024-01-16T16:56:23-03:00\" level=warning msg=\"[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.17.145.181]\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Starting container [file-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Successfully started [file-deployer] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Waiting for [file-deployer] container to exit on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:23-03:00\" level=info msg=\"Waiting for [file-deployer] container to exit on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Removing container [file-deployer] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[remove/file-deployer] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[reconcile] Reconciling cluster state\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"[reconcile] This is newly generated cluster\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Pre-pulling kubernetes images\"", "time=\"2024-01-16T16:56:24-03:00\" level=info msg=\"Pulling image [rancher/hyperkube:v1.24.10-rancher4] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Kubernetes images pulled successfully\"", "time=\"2024-01-16T16:57:39-03:00\" level=info msg=\"[etcd] Building up etcd plane..\"", "time=\"2024-01-16T16:57:39-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Starting container [etcd-fix-perm] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Successfully started [etcd-fix-perm] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Waiting for [etcd-fix-perm] container to exit on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:40-03:00\" level=info msg=\"Container [etcd-fix-perm] is still running on host [172.17.145.181]: stderr: [], stdout: []\"", "time=\"2024-01-16T16:57:41-03:00\" level=info msg=\"Removing container [etcd-fix-perm] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:41-03:00\" level=info msg=\"[remove/etcd-fix-perm] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:41-03:00\" level=info msg=\"Pulling image [rancher/mirrored-coreos-etcd:v3.5.4] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Image [rancher/mirrored-coreos-etcd:v3.5.4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Starting container [etcd] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Successfully started [etcd] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Running rolling snapshot container [etcd-snapshot-once] on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"Starting container [etcd-rolling-snapshots] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:57:57-03:00\" level=info msg=\"[etcd] Successfully started [etcd-rolling-snapshots] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:02-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Starting container [rke-bundle-cert] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:03-03:00\" level=info msg=\"[certificates] Successfully started [rke-bundle-cert] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Waiting for [rke-bundle-cert] container to exit on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:03-03:00\" level=info msg=\"Container [rke-bundle-cert] is still running on host [172.17.145.181]: stderr: [], stdout: []\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Removing container [rke-bundle-cert] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:04-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[etcd] Successfully started [rke-log-linker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:05-03:00\" level=info msg=\"[etcd] Successfully started etcd plane.. Checking etcd cluster health\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[etcd] etcd host [172.17.145.181] reported healthy=true\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[controlplane] Building up Controller Plane..\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Finding container [service-sidekick] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"Starting container [kube-apiserver] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[controlplane] Successfully started [kube-apiserver] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:06-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-apiserver] on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:11-03:00\" level=info msg=\"[healthcheck] service [kube-apiserver] on host [172.17.145.181] is healthy\"", "time=\"2024-01-16T16:58:11-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:11-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"Starting container [kube-controller-manager] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[controlplane] Successfully started [kube-controller-manager] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:12-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:17-03:00\" level=info msg=\"[healthcheck] service [kube-controller-manager] on host [172.17.145.181] is healthy\"", "time=\"2024-01-16T16:58:17-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:17-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"Starting container [kube-scheduler] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[controlplane] Successfully started [kube-scheduler] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:18-03:00\" level=info msg=\"[healthcheck] Start Healthcheck on service [kube-scheduler] on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:23-03:00\" level=info msg=\"[healthcheck] service [kube-scheduler] on host [172.17.145.181] is healthy\"", "time=\"2024-01-16T16:58:23-03:00\" level=info msg=\"Image [rancher/rke-tools:v0.1.88] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[controlplane] Successfully started [rke-log-linker] container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Removing container [rke-log-linker] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[remove/rke-log-linker] Successfully removed container on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[controlplane] Successfully started Controller Plane..\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating rke-job-deployer ServiceAccount\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] rke-job-deployer ServiceAccount created successfully\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating system:node ClusterRoleBinding\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] system:node ClusterRoleBinding created successfully\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Successfully Deployed state file at [/home/magma/magma-deployer/rke/cluster.rkestate]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[state] Saving full cluster state to Kubernetes\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[state] Successfully Saved full cluster state to Kubernetes ConfigMap: full-cluster-state\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[worker] Building up Worker Plane..\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Finding container [service-sidekick] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"[sidekick] Sidekick container already created on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Image [rancher/hyperkube:v1.24.10-rancher4] exists on host [172.17.145.181]\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #1\"", "time=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #2\"", "time=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"", "time=\"2024-01-16T16:58:24-03:00\" level=info msg=\"Starting container [kubelet] on host [172.17.145.181], try #3\"", "time=\"2024-01-16T16:58:24-03:00\" level=warning msg=\"Can't start Docker container [kubelet] on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount\"", "time=\"2024-01-16T16:58:24-03:00\" level=fatal msg=\"[workerPlane] Failed to bring up Worker Plane: [Failed to start [kubelet] container on host [172.17.145.181]: Error response from daemon: path /var/lib/kubelet is mounted on / but it is not a shared mount]\""]}