Downscaling of RKE2 Clusters failed

[mueller-tobias]

Description

Rancher support the up- and downscaling of new nodes via ui. I can successfully scale up ionoscloud clusters, but i cant' downscale those cluster from e.g. 5 workers to 3 worker. After downscaling the cluster start the Reconciling phase but dosn't start removing the node. In the rancher i logs i see the errors below:

Expected behavior

When i scale down a cluster the nodes are removed properly

Environment

Docker Machine Driver Ionos Cloud version:

7.0.0-rc.2

Configuration Files

How to Reproduce

• Create a 3 Node Cluster with a DHCP IP from a Customer DHCP
• Wait until the Cluster is active
• Delete 1 Node

Cluster is first reconcinling, but dosn't remove the node. The deleted nodes hangs with the state deleting. No Removal of the VM on the IONOS Datacenter

Error and Debug Output

2023-08-03T12:30:47.130552084Z 2023/08/03 12:30:47 [INFO] EnsureSecretForServiceAccount: waiting for secret [rke2-test-bootstrap-template-ws4hz-machine-bootstrap-tokenhvp4b] to be populated with token
2023/08/03 12:30:47 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:47.193569342Z 2023/08/03 12:30:47 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:47.226394127Z 2023/08/03 12:30:47 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:47.240621085Z 2023/08/03 12:30:47 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:52.244323788Z 2023/08/03 12:30:52 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:52.263875499Z 2023/08/03 12:30:52 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:52.444868606Z 2023/08/03 12:30:52 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for plan to be applied
2023-08-03T12:30:54.093823235Z 2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kube-apiserver, kubelet
2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kube-apiserver, kubelet
2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:54.191788063Z 2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:54.230651651Z 2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:54.311379071Z 2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:54.403134187Z 2023/08/03 12:30:54 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:57.482977471Z 2023/08/03 12:30:57 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:57.502529244Z 2023/08/03 12:30:57 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023/08/03 12:30:59 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:30:59.964012812Z 2023/08/03 12:30:59 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:00.009693458Z 2023/08/03 12:31:00 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:00.032407431Z 2023/08/03 12:31:00 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:00.148964440Z 2023/08/03 12:31:00 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:00.218492028Z 2023/08/03 12:31:00 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:00.342640750Z 2023/08/03 12:31:00 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: configuring bootstrap node(s) rke2-test-pool1-cbc7d49dd-489z4: waiting for probes: kubelet
2023-08-03T12:31:04.128966564Z 2023/08/03 12:31:04 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: rke2-test-pool1-cbc7d49dd-489z4
2023/08/03 12:31:06 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:06.245667651Z 2023/08/03 12:31:06 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:06.264679876Z 2023/08/03 12:31:06 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023/08/03 12:31:06 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023/08/03 12:31:06 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:07.584267766Z 2023/08/03 12:31:07 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:07.661778085Z 2023/08/03 12:31:07 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:10.483210484Z 2023/08/03 12:31:10 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:10.508967666Z 2023/08/03 12:31:10 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023/08/03 12:31:10 [INFO] [planner] rkecluster fleet-default/rke2-test: waiting: non-ready bootstrap machine(s) rke2-test-pool1-cbc7d49dd-489z4: Node condition Ready is False.
2023-08-03T12:31:52.187277802Z 2023/08/03 12:31:52 [ERROR] error syncing 'fleet-default/rke2-test-pool1-c9cd06f9-2kncg': handler machine-provision-remove: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes rke2-test-pool1-c9cd06f9-2kncg), requeuing
2023/08/03 12:32:52 [ERROR] error syncing 'fleet-default/rke2-test-pool1-c9cd06f9-2kncg': handler machine-provision-remove: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes rke2-test-pool1-c9cd06f9-2kncg), requeuing
2023/08/03 12:33:52 [ERROR] error syncing 'fleet-default/rke2-test-pool1-c9cd06f9-2kncg': handler machine-provision-remove: the server was unable to return a response in the time allotted, but may still be processing the request (get nodes rke2-test-pool1-c9cd06f9-2kncg), requeuing

[mueller-tobias]

Seem's like the UI Extension didn't install properly. I reinstalled the node driver and the extension on a fresh rancher instance and could sucessfully remove the node on a new build rke2 cluster