brandond
commented
2 years ago Aug 3 10:06:00 production-island10-overcloud-controlplane-3 rke2[2020]: time="2022-08-03T10:06:00Z" level=info msg="certificate CN=production-island10-overcloud-worker-48 signed by CN=rke2-server-ca@1659518912: notBefore=2022-08-03 09:28:32 +0000 UTC notAfter=2023-08-03 10:06:00 +0000 UTC"
This log entry is for the RKE2 server issuing the agent node's kubelet certificates. These certificates are renewed every time the agent starts. Since you see it being issued repeatedly, I suspect that rke2 agent is crashing, restarting, and requesting new certificates, in a loop. What do the logs from the agent node show?
I might also suggest that you not try to add all the nodes in parallel, at the exact same time. There's no good reason to put that much load on the system at once. Stagger the nodes by 15 to 30 seconds at least so that you don't overwhelm the control-plane, datastore, and image registry.
vriabyk
niusmallnan
stale[bot]
Sea-you
Environmental Info: RKE2 Version:
rke2 version v1.23.6+rke2r2 (40d712e5081ac87e30e8f328f738130acf2c31f8) go version go1.17.5b7 also we tried it on rke2 version v1.22.7+rke2r2
Rancher Version: v2.6.6
rancher2 tf provider version: 1.24.0
Node(s) CPU architecture, OS, and Version:
Custom cluster type, Kubevirt Ubuntu VMs: Linux 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
controlplane nodes: cpu = "8" memory = "32G"
etcd nodes: cpu = "4" memory = "16G"
worker nodes: cpu = "6" memory = "32G"
JFYI, we don't observe any highloads or any spikes on the nodes during deployment and afterwards.
Cluster Configuration:
Deploying via Rancher terraform provider: 3 controlplane nodes, 3 etcd and 100 workers
Describe the bug:
When we try to deploy rke2 cluster using Rancher and rancher2 terraform provider with 100 workers or more, the deployment gets stuck on configuring worker nodes:
It manages to spawn and configure etcd, controlplane nodes and even add a few workers (looks like random nodes):
But all other worker nodes are stuck in configuring state in rancher. If we restart rke2 server on the node which is defined in rke2 agent configuration file, it may register several new workers.
For example, If we try to set 50 worker nodes - the deployment always completes successfully. Also we tried to add workers by portions, for example by 50 workers. It worked also fine and we were able to deploy 300 workers which is our target. We didn't try to deploy 60, 70 or 80 workers, so don't really know what actually the critical number to be deployed at one time in parallel.
Steps To Reproduce:
Deploy rke2 cluster with 100 or more workers in parallel via Rancher. We use rancher2 tf provider and remote exec to run curl from each node to trigger system-agent-install.sh, but I don't think this is the problem. You can add workers in any other method, the only requirement is to add such number of workers at one time(or almost at one time), so rke2 server should process them in parallel.
Expected behavior:
The cluster is getting deployed and becomes Active state in Rancher, all worker nodes are registered. Our target - at least 300 worker to be deployed in parallel.
Actual behavior:
As mentioned above, the deployment gets stuck on configuring the most part of worker nodes.
Additional context / logs:
At rke2 agent side, there are lots of records like this:
At rke2 server side logs for the node are:
All rke2 agents try to connect to the first launched rke2 server node (in this case production-island10-overcloud-controlplane-3) and we see a lot of connections to 9345 port:
For me it looks like rke2 server generates some kind of "registration" certificate for rke2 agent which is valid only 20 seconds (looking at rke2 server logs above, notAfter field changes on 20 sec every time). My feeling is when we deploy a huge number of nodes in parallel (like 100 or more), rke2 server doesn't manage to reply in time to rke2 agent requests and certificate gets expired and the client times out.
Any help or recommendation will be much appreciated!