[TEST] v1.3.2 release testing - Four node upgrade with bonded nics

[TachunLin]

What's the test to develop? Please describe

Perform the four nodes upgrade test with bonded nics for v1.3.2 release testing

Prerequisite and dependency of test

1. VLAN 1 network on mgmt and 1 network on bonded NICs
2. 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
3. 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped
4. Create a new storage class apart from default one. Use the new storage class for some basic operations.
5. Import to Rancher and create an RKE2 guest cluster. provisioned on Harvester VM before the upgrade.
6. Verify DHCP load balancer service
7. Install Harvester CSI Driver (prior to latest version)
8. Create a new Harvester PVC for nginx deployment
9. Upgrade Rancher

Upgrade Test Path:

• H1.3.1 + R2.7.14 (K1.27) -> H1.3.2 + R2.7.15 (K1.27)

OS:

• SLES 15 SP6 (sles@ipv4)
• SLES 15 SP4 (sles@ipv4)
• Ubuntu 24.04 (ubuntu@ipv4)
• SLE Micro 6 [sles], see QA Message

Upgrade Test Check

1. Can correctly upgrade from v1.3.1 to v1.3.2-rc

Post Upgrade Check

1. Dependencies Check
   1. dependency_charts
   2. Versions of all the crds
2. Virtual machines are in same state as before and accessible.
3. Restore the backups, check the data
4. Image and volume status
5. Monitoring chart status
6. VM operations are highlighted and working fine.
7. Upgrade RKE2 guest cluster
8. Upgrade cloud provider and CSI driver
9. Verify DHCP load balancer service and create a new Harvester PVC
10. Shutting off VM and then restarting VM

[TachunLin]

Four nodes upgrade test on Ubuntu 24.04 from v1.3.1 to v1.3.2-rc2

Test environment

Harvester: v1.3.1 upgrade to v1.3.2-rc2 Rancher: v2.7.14 (Prime) RKE2 version: v1.27.16 OS: Ubuntu 24.04

Prerequisite and dependency setup

${\textsf{✅}}$ VLAN 1 network on mgmt and 1 network on bonded NICs $~~$


${\textsf{✅}}$ 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped $~~$

 ``` ubuntu@ubuntu2404-running:~$ sha512sum running.txt 09318b9fe73630d941a2f4fc9e7509d39565391d555abf20000634f07e357c1c67262bea1d18fe9078db0b37292424f16f56cb90f7f2714e9c2ce22b3595d6cb running.txt ``` ``` ubuntu@ubuntu2404-stopped:~$ sha512sum stopped.txt e4b11c7e5dbcff75eb76a2245f3ecb954ca8fe88ebeb4c263b29e6f4065bf9591d5cb09c4f5385a2cc514ca80233dd496ba45aee79ef2b1ca0c7ff3efb33d94d stopped.txt ```

${\textsf{✅}}$ 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped $~~$


${\textsf{✅}}$ Create a new storage class apart from default one. Use the new storage class for some basic operations. $~~$


${\textsf{✅}}$ Import to Rancher 2.7.14 and create an RKE2 v1.27.16 guest cluster. provisioned on Harvester VM before the upgrade. $~~$

 

${\textsf{✅}}$ Verify cloud provider and csi-driver version and running $~~$

1. cloud provider: 0.2.4 2. csi-driver: 0.1.17 

${\textsf{✅}}$ Verify DHCP load balancer service $~~$


${\textsf{✅}}$ Create a new Harvester PVC for nginx deployment $~~$

  

Upgrade from v1.3.1 to v1.3.2-rc2

${\textsf{✅}}$ Successfully upgrade to v1.3.2-rc2 $~~$

* Remember to apply this workaround https://github.com/harvester/harvester/issues/5756#issuecomment-2112855477 

Post Upgrade Check

${\textsf{✅}}$ Dependency check $~~$

Chart: https://github.com/harvester/harvester/tree/v1.3.2-rc2/deploy/charts/harvester/dependency_charts Values: https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/values.yaml - csi-snapshotter

$\color{green}{\textsf{PASS}}$ $~~$

- tag: v6.3.3 https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/dependency_charts/csi-snapshotter/values.yaml - Harvester: csi-snapshotter 

- kubevirt-operator

$\color{green}{\textsf{PASS}}$ $~~$

- tag: 1.1.1-150500.8.18.1 https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/values.yaml - Harvester: virt-operator (deployment) 

- kubevirt

$\color{green}{\textsf{PASS}}$ $~~$

- tag: 1.1.1-150500.8.18.1 https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/values.yaml - Harvester: kubevirt (crd) 

- snapshot-validation-webhook

$\color{green}{\textsf{PASS}}$ $~~$

- tag: 6.3.3 https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/values.yaml - Harvester: harvester-snapshot-validation-webhook- (deployment) 

- whereabouts

$\color{green}{\textsf{PASS}}$ $~~$

- tag: 0.6.3 https://github.com/harvester/harvester/blob/v1.3.2-rc2/deploy/charts/harvester/values.yaml - Harvester: harvester-whereabouts- (pod) 

${\textsf{✅}}$ Versions of all the crds $~~$

https://github.com/harvester/harvester/tree/v1.3.2-rc2/deploy/charts/harvester-crd/templates - addons.harvesterhci.io

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_addons.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_addons.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- keypairs.harvesterhci.io

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_keypairs.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_keypairs.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_preferences

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_preferences.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_preferences.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_settings

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_settings.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_settings.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_supportbundles

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_supportbundles.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_supportbundles.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_upgradelogs

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_upgradelogs.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_upgradelogs.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_upgrades

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_upgrades.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_upgrades.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_versions

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_versions.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_versions.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_virtualmachinebackups

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinebackups.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinebackups.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_virtualmachineimages

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachineimages.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachineimages.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_virtualmachinerestores

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinerestores.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinerestores.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_virtualmachinetemplates

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplates.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplates.yaml Observation: v1beta1 Expectation: v1beta1 True ```

- harvesterhci.io_virtualmachinetemplateversions

$\color{green}{\textsf{PASS}}$ $~~$

https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplateversions.yaml ``` harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplateversions.yaml Observation: v1beta1 Expectation: v1beta1 True ```

${\textsf{✅}}$ Virtual machines are in same state as before and accessible $~~$


${\textsf{✅}}$ Restore the backups, check the data $~~$

1. Restore the running backup vm to new, the data can be restored and exists  2. Restore the stored backup vm to new, the data can be restored and exists 

${\textsf{✅}}$ Image and volume status $~~$

 

${\textsf{✅}}$ Monitoring chart status $~~$

   

${\textsf{✅}}$ VM operations are highlighted and working fine $~~$

* Start * Stop * Restart * Soft reboot * Delete * Clone * Migrate * Add volume

${\textsf{✅}}$ Upgrade Rancher to v2.7.15 $~~$

1. Can successfully upgrade Rancher to v2.7.15 and the RKE2 cluster is running 

${\textsf{✅}}$ Upgrade RKE2 guest cluster to v1.27.6+rke2r2 $~~$

1. Can successfully upgrade the RKE2 cluster to `v1.27.6+rke2r2`  

${\textsf{✅}}$ Upgrade cloud provider and CSI driver $~~$

1. Automatically upgrade cloud provider to `v0.2.4` and csi-driver to `0.2.16` 

${\textsf{✅}}$ Create a new Harvester PVC $~~$

1. Create a new Harvester PVC for nginx deployment   

${\textsf{✅}}$ Verify DHCP load balancer service $~~$

1. Verify DHCP load balancer service