[TEST] v1.3.2 Release Testing: Two nodes upgrade

[albinsun]

What's the test to develop? Please describe

Three node upgrade w/ 2 MGMT/Default Nodes & 1 Witness Node

• Uprade Path [5]: H1.3.1 + R2.7.15 (K1.27) -> H1.3.2 + R2.8.6 (K1.28)
• OS [4, 5, 6, 7]:
  • SLE Micro 6 [sles], see QA Message
  • Windows 11 enterprise ``(no rke2 testing needed)
  • Windows 2022 (no rke2 testing needed)
  • RHEL 9.4 (cloud-user@ipv4)

Prerequisite and dependency of test

1. Setup three nodes cluster w/ witness node
2. VLAN 1 network on mgmt and 1 network on other NICs
3. 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
4. 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped
5. Create a new storage class apart from default one. Use the new storage class for some basic operations.
6. Import to Rancher and create an RKE2 guest cluster provisioned on Harvester VM before the upgrade.
7. Deploy Harvester cloud provider to RKE1 Cluster (prior to latest version)
8. Verify DHCP load balancer service
9. Install Harvester CSI Driver (prior to latest version)
10. Create a new Harvester PVC for nginx deployment
11. Upgrade Rancher
12. Upgrade Harvester

Describe the items of the test development (DoD, definition of done) you'd like

1. Dependencies Check
   • dependency_charts
   • Versions of all the crds
2. Virtual machines are in same state as before and accessible.
3. Restore the backups, check the data
4. Image and volume status
5. Monitoring chart status
6. VM operations are highlighted and working fine.
7. Import Harvester into a Rancher cluster
8. Add a node after the upgrade
9. Upgrade the RKE2 guest cluster to 1.27 or above
10. Upgrade cloud provider and CSI driver
11. Verify DHCP load balancer service and create a new Harvester PVC
12. Shutting off VM and then restarting VM

Additional context

[albinsun]

Prerequisite and dependency of test

1. :green_circle: Setup 2 nodes v1.3.1

   * hosts  * node-0  * node-1 
2. :green_circle: Create new storage class w/ replica 2 and set as default

   
3. :green_circle: VLAN 1 network on mgmt and 1 network on other NICs

   * Cluster Network  * VM Network 
4. :yellow_circle: 2 Virtual machines with data and md5sum computed- 2 running, 2 stopped

   * Images  * VMs  * RHEL 9.4 (cloud-user@ipv4): Running, ens5-vlan1, extra disk  * SLE Micro 6 [sles]: Stopped, mgmt-vlan1  * Windows 11 enterprise: Running, mgmt-vlan1 (TPM, EFI and Secure Boot) 2C / 4G / 64G  * Windows 2022 (no rke2 testing needed): Stopped, ens5-vlan1 (TPM, EFI and Secure Boot) * Insufficient Storage 

   :warning: Temporarily skip windows VMs as insufficient local storage space. Will test seperately on lab env.

5. :yellow_circle: 2 VM backup, snapshots - 1 when VM is running and 1 when VM is stooped

   * Snapshot  * Backup 

   harvester/harvester/issues/6497

6. :green_circle: Import to Rancher and create an RKE2 (v1.27) guest cluster provisioned on Harvester VM before the upgrade.

   * Import Harvester to Rancher  * Provision RKE2 guest cluster 
7. :green_circle: Deploy Harvester cloud provider to RKE2 Cluster (prior to latest version)

   * rke2-rhel9 
8. :green_circle: Install Harvester CSI Driver (prior to latest version)

   * rke2-rhel9  * rke2-slm6 
9. :green_circle: Create a new Harvester PVC for nginx deployment

   * Default storage class * rke2-rhel9  * rke2-slm6  * Create PVC * rke2-rhel9  * rke2-slm6 
10. :green_circle: Verify DHCP load balancer service

    * rke2-rhel9   * rke2-slm6  
11. :green_circle: Upgrade to Rancher v2.8.6

    * rke2-rhel9 * RKE2 cluster still running  * nginx still works  * rke2-slm6 * upgrade rancher  * RKE2 cluster still running  * nginx still works 

    Ref. https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades

12. :green_circle: Upgrade to RKE2 v1.28

    * rke2-rhel9 * upgrade RKE2 version  * RKE2 cluster still running  * nginx still works   * rke2-slm6 * upgrade RKE2 version  * RKE2 cluster still running  * nginx still works 

[albinsun]

Upgrade to harvester-v1.3.1 to harvester-v1.3.2-rc2

1. :green_circle: Prepare an airgapped upgrade

   

   Ref. https://docs.harvesterhci.io/v1.3/upgrade/index/#prepare-an-air-gapped-upgrade

2. :green_circle: Trigger Upgrade

   
3. :green_circle: Successfully upgrade

   

   :warning: Has chance to hit harvester/harvester/issues/6432

[albinsun]

Post-upgrade Checks

1. :green_circle: Dependencies Check

   ``` $ python3 check_chart_version.py -cac -ip 192.168.0.30 ... For /home/rancher/harvester_yamls/crd-kubevirt.yaml - Observation: v1 v1alpha3 Expectation: v1 v1alpha3 True For /home/rancher/harvester_yamls/harvesterhci.io_addons.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_keypairs.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_preferences.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_settings.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_supportbundles.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_upgradelogs.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_upgrades.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_versions.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_virtualmachinebackups.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_virtualmachineimages.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_virtualmachinerestores.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_virtualmachinetemplates.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/harvesterhci.io_virtualmachinetemplateversions.yaml - Observation: v1beta1 Expectation: v1beta1 True For /home/rancher/harvester_yamls/volumesnapshotclasses.yaml - Observation: v1 v1beta1 Expectation: v1 v1beta1 True For /home/rancher/harvester_yamls/volumesnapshotcontents.yaml - Observation: v1 v1beta1 Expectation: v1 v1beta1 True For /home/rancher/harvester_yamls/volumesnapshots.yaml - Observation: v1 v1beta1 Expectation: v1 v1beta1 True For /home/rancher/harvester_yamls/whereabouts.cni.cncf.io_ippools.yaml - Observation: v1alpha1 Expectation: v1alpha1 True For /home/rancher/harvester_yamls/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml - Observation: v1alpha1 Expectation: v1alpha1 True ```
2. :green_circle: Virtual machines are in same state as before and accessible.

   
3. :green_circle: Restore the backups, check the data

   Restore New  Restore Replace  
4. :green_circle: Image and volume status

   * Image  * Volume 
5. :green_circle: VM operations are highlighted and working fine.

   - [x] Restart - [x] Soft Reboot - [x] Migrate to default node - [x] Add Volume
6. :green_circle: RKE2 guest cluster still works

   * cluster  * LB 
7. :green_circle: Upgrade cloud provider and CSI driver

    * cloud-provider  * csi-driver 
8. :green_circle: Verify DHCP load balancer service and create a new Harvester PVC

   * LB  * New PVC 
9. :green_circle: Shutting off VM and then restarting VM

   - [x] Stop - [x] Start
10. :green_circle: Add a (Default) node after the upgrade

    

[albinsun]

Windows VMs will be test seperately.

[albinsun]

FYI, try reproduce harvester/harvester#6432 3 times but not hit

Note that we can refer workaround in harvester/docs/pull/635 if hit again.

[albinsun]

Update RHEL9 result on harvester-v1.3.2 + rancher-v2.7.15 (v1.27) -> rancher-v2.8.6 (v1.28)

1. :green_circle: Deploy Harvester cloud provider to RKE2 Cluster (prior to latest version)

   * rke2-rhel9 
2. :green_circle: Install Harvester CSI Driver (prior to latest version)

   * rke2-rhel9  * rke2-slm6 
3. :green_circle: Create a new Harvester PVC for nginx deployment

   * Default storage class * rke2-rhel9  * rke2-slm6  * Create PVC * rke2-rhel9  * rke2-slm6 
4. :green_circle: Verify DHCP load balancer service

   * rke2-rhel9   * rke2-slm6  
5. :green_circle: Upgrade to Rancher v2.8.6

   * rke2-rhel9 * RKE2 cluster still running  * nginx still works  * rke2-slm6 * upgrade rancher  * RKE2 cluster still running  * nginx still works 

   Ref. https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades

6. :green_circle: Upgrade to RKE2 v1.28

   * rke2-rhel9 * upgrade RKE2 version  * RKE2 cluster still running  * nginx still works   * rke2-slm6 * upgrade RKE2 version  * RKE2 cluster still running  * nginx still works