Closed albinsun closed 2 months ago
Another reproduce, this time both 2 machines stuck at Reconciling
over 30 mins.
But harvester side did scale down to 1 VM
Support bundle supportbundle_ScaleDownFail2_2023-08-01T14-18-30Z.zip
Waiting for plan to be applied
, the other shows Waiting for probes: etcd, kube-apiserver, kube-controller-manager
FYI
Same steps are OK if using Rancher v2.7.4
(RKE2: v1.25.11+rke2r1)
supportbundle_ScaleDownOk-r274_2023-08-01T16-30-10Z.zip
@albinsun This could be related to https://github.com/rancher/rancher/issues/42034. Could you check with Rancher 2.7-head whenever get a chance?
FYI, test pass with Rancher v2.7-head
(d8a2bca)
v1.2.0-rc4
v2.7-head
(d8a2bca)Scale Pool Up ${~~~\color{green}\textsf{V}}$
Scale Pool Down ${~~~\color{green}\textsf{V}}$
Thanks @albinsun for checking this, we can close this for now as this is solved with Rancher upcoming release 2.7.6.
cc: @guangbochen @bk201
Reopen since fail to scale down with Harvester v1.2.0-rc5
+ Rancher v2.7.6-rc4
Rkecontrolplane was already initialized but no etcd machines exist that have plans, indicating the etcd plane has been entirely replaced. Restoration from etcd snapshot is required.
v1.2.0-rc5
v1.25.9+rke2r1
v2.7.6-rc4
v1.26.7+rke2r1
Check Workload and Service Discovery ${~~~\color{gray}\textsf{?}}$
By following the testing plan on https://github.com/rancher/rancher/issues/42121#issuecomment-1656310669, it seems scale down the master machine pool from 2 to 1 works fine in my test(Harvester v1.2.0-rc5 + Rancher v2.7.6-rc4)
Test k3s-rancher (v2.7.6-rc4
), scale up and down twice, looks fine.
v1.2.0-rc5
v1.25.9+rke2r1
v2.7.6-rc4
v1.26.7+k3s1
I tested with vcluster-rancher (Rancher v2.7.6-rc4). Scale up and down twice and it looks good.
Test rke2-rancher (v2.7.6-rc5
), scale up and down twice, fail to scale down in the 2nd round.
In the failed case, we notice that the machine name is neither of 2 before scaled down.
v1.2.0-rc5
v1.25.9+rke2r1
v2.7.6-rc5
v1.26.7+rke2r1
Upstream issue: https://github.com/rancher/rancher/issues/42582
Also hit in K3s cluster, sysrem delete both 2 original nodes and create a new one, then status stuck in Updating
(rkecontrolplane was already initialized but no etcd machines exist that have plans, indicating the etcd plane has been entirely replaced. Restoration from etcd snapshot is required.
)
We encounter similar issue on Rancher v2.7.9 with Harvester v1.1.3-rc1
While scaling down the RKE2 cluster from 3 nodes to 2 nodes, the first node stuck in Reconcilling
state
Attached the Harvester support bundle, Rancher instance logs for more information
Support bundle rke2-failed-scale-down_2023-12-11T03-52-45Z.zip
Rancher logs rancher.log
https://github.com/rancher/rancher/issues/42582 seems fixed in Rancher 2.9.0.
The upstream PR https://github.com/kubernetes-sigs/cluster-api/pull/10431 was included in CAPI v1.6.6. In Rancher v2.9.0-alpha5, the default CAPI version is v1.6.6. I have confirmed the issue can't be reproduced with the version.
[ ] If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted? The HEP PR is at:
[ ] Where is the reproduce steps/test steps documented? The reproduce steps/test steps are at:
[ ] Is there a workaround for the issue? If so, where is it documented? The workaround is at:
[ ] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*
)?
The PR is at:
[ ] Does the PR include the explanation for the fix or the feature?
[ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart? The PR for the YAML change is at: The PR for the chart change is at:
[ ] If labeled: area/ui Has the UI issue filed or ready to be merged? The UI issue/PR is at:
[ ] If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged? The documentation/KB PR is at:
[ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
[ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility
?
The compatibility issue is filed at:
Automation e2e test issue: harvester/tests#1348
also note the fix is not going to be backported to Rancher 2.8.x https://github.com/rancher/rancher/issues/42582#issuecomment-2197670273
Test OK on v1.3.1
+ rancher-v2.9.0-alpha7
with RKE2 v1.28.10+rke2r1
, close as fixed.
v1.3.1
Auto
v2.9.0-alpha7
v1.28.11+k3s1
) on QEMU/KVM (2C/4G)v1.28.10+rke2r1
^11 -> 2 -> 1
2 -> 3 -> 2
Description
After scale up RKE2 cluster from 1 to 2 VMs, Fail to scale down back.
Environment
v1.2.0-rc4
v2.7.5
v1.26.6+rke2r1
To Reproduce
Import Harvester into Rancher
![image](https://github.com/harvester/harvester/assets/2773781/60a4cea4-771e-4150-aeeb-b9ccf1b8e0a8)[H] Upload image "focal-server-cloudimg-amd64.img"
![image](https://github.com/harvester/harvester/assets/2773781/4200846d-8134-40ca-a905-313d7c8cf73a)[H] Create VM Network "mgmt-vlan1"
![image](https://github.com/harvester/harvester/assets/2773781/a6ccf394-3e7e-4b35-bb68-de1e66816161)[R] Create RKE2 Harvester node driver cluster "rke2-hvst-r275" (Takes ~20m)
![image](https://github.com/harvester/harvester/assets/2773781/26ca476a-4f1e-4244-9939-0b20adef1623)focal-server-cloudimg-amd64.img
mgmt-vlan1
[R] Check rke2-hvst-r275 is Active
![image](https://github.com/harvester/tests/assets/2773781/b943e1d6-fc28-4e62-9691-5f1018539321)[H] Check 1 VM is Running and Harvester cluster is healthy
![image](https://github.com/harvester/tests/assets/2773781/9d8c149c-7dfb-4d15-a01a-4af98178699f) ![image](https://github.com/harvester/tests/assets/2773781/dc056c08-9d8a-4989-8150-b4d5fb15692e)[R] Scale up rke2-hvst-r275 (Takes ~20m)
Using the "+" button ![image](https://github.com/harvester/tests/assets/2773781/b943e1d6-fc28-4e62-9691-5f1018539321)[R] Check rke2-hvst-r275 is Active and has 2 machines in pool
![image](https://github.com/harvester/tests/assets/2773781/9745f38d-4caa-4fe2-923c-8140876fc3e4) ![image](https://github.com/harvester/tests/assets/2773781/de93d77e-4601-4821-b786-7d4525849f66)[H] Check 2 VM is Running and Harvester cluster is healthy
![image](https://github.com/harvester/tests/assets/2773781/f48e35a4-92cd-4bf3-bd89-429b076f3c17) ![image](https://github.com/harvester/tests/assets/2773781/f1293379-272e-4f19-b782-5facf6cbe8c4)[R] Scale down rke2-hvst-r275 ${~~~\color{red}\textsf{X, do scale-downed to 1 VM on both sides (H and R), but the machine status on Rancher stuck in Updating}}$
* Status stuck in `Updating` ![image](https://github.com/harvester/harvester/assets/2773781/8294e9ab-ebe4-451d-84a5-0d58923ce85d) ![image](https://github.com/harvester/harvester/assets/2773781/0b7f9503-a9f5-4f63-8780-378eaafe6667) * Error Messge ``` Rkecontrolplane was already initialized but no etcd machines exist that have plans, indicating the etcd plane has been entirely replaced. Restoration from etcd snapshot is required. ``` ![image](https://github.com/harvester/harvester/assets/2773781/7b871d12-ab0f-457b-a7fd-54786edb43f4) * Provisioning Log ![image](https://github.com/harvester/harvester/assets/2773781/026519ac-c832-4ac9-915e-edfebde5c7f1)Expected Behavior
Support Bundle
supportbundle_ScaleDownFail_2023-08-01T11-19-57Z.zip
Note
v2.7.4
, see comment below.Upstream issue: https://github.com/rancher/rancher/issues/42582