[TEST] v1.3.2 Release Testing: Three node upgrade w/ 2 MGMT/Default Nodes & 1 Witness Node

[albinsun]

What's the test to develop? Please describe

Three node upgrade w/ 2 MGMT/Default Nodes & 1 Witness Node

• Uprade Path [3]: H1.3.1 + R2.8.5 (k1.28) -> H1.3.2 + R2.9.0 (K1.29) -> (K1.30)
• OS [1, 2, 3, 7]:
  • SLES 15 SP6 (sles@ipv4)
  • SLES 15 SP4 (sles@ipv4)
  • Ubuntu 24.04 (ubuntu@ipv4)
  • RHEL 9.4 (cloud-user@ipv4)

Prerequisite and dependency of test

1. Setup three nodes cluster w/ witness node
2. VLAN 1 network on mgmt and 1 network on other NICs
3. 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
4. 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped
5. Create a new storage class apart from default one. Use the new storage class for some basic operations.
6. Import to Rancher and create an RKE2 guest cluster provisioned on Harvester VM before the upgrade.
7. Deploy Harvester cloud provider to RKE1 Cluster (prior to latest version)
8. Verify DHCP load balancer service
9. Install Harvester CSI Driver (prior to latest version)
10. Create a new Harvester PVC for nginx deployment
11. Upgrade Rancher
12. Upgrade Harvester

Describe the items of the test development (DoD, definition of done) you'd like

1. Dependencies Check
   • dependency_charts
   • Versions of all the crds
2. Virtual machines are in same state as before and accessible.
3. Restore the backups, check the data
4. Image and volume status
5. Monitoring chart status
6. VM operations are highlighted and working fine.
7. Import Harvester into a Rancher cluster
8. Add a node after the upgrade
9. Upgrade the RKE2 guest cluster to 1.27 or above
10. Upgrade cloud provider and CSI driver
11. Verify DHCP load balancer service and create a new Harvester PVC
12. Shutting off VM and then restarting VM

Additional context

[albinsun]

Prerequisite and dependency of test

1. :green_circle: Setup three nodes v1.3.1 cluster w/ witness node

   * hosts  * node-0  * node-1  * node-2 
2. :green_circle: Create new storage class w/ replica 2 and set as default

   
3. :green_circle: VLAN 1 network on mgmt and 1 network on other NICs

   * Cluster Network   * VM Network 
4. :green_circle: 4 Virtual machines with data and md5sum computed- 2 running, 2 stopped

   * Images  * VMs  * SLES 15 SP6 (sles@ipv4): Running, mgmt-vlan1, extra disk.  * SLES 15 SP4 (sles@ipv4): Stopped, ens5-vlan1  * Ubuntu 24.04 (ubuntu@ipv4): Running, ens5-vlan1  * RHEL 9.4 (cloud-user@ipv4): Stopped, mgmt-vlan1, extra disk. 
5. :green_circle: 2 VM backup, snapshots - 1 when VM is running and 1 when VM is stooped

   * SLES 15 SP6 (sles@ipv4): Running - Backup * SLES 15 SP4 (sles@ipv4): Stopped - Snapshot * Ubuntu 24.04 (ubuntu@ipv4): Running - Snapshot * RHEL 9.4 (cloud-user@ipv4): Stopped - Backup * Snapshot  * Backup 
6. :green_circle: Import to Rancher and create an RKE2 (v1.28) guest cluster provisioned on Harvester VM before the upgrade.

   * Import Harvester to Rancher  * Provision RKE2 guest cluster 
7. :green_circle: Deploy Harvester cloud provider to RKE2 Cluster (prior to latest version)

   
8. :green_circle: Install Harvester CSI Driver (prior to latest version)

   
9. :green_circle: Create a new Harvester PVC for nginx deployment

   * Default storage class  * Create PVC 
10. :green_circle: Verify DHCP load balancer service

    * Nginx deployment  * DHCP load balancer  
11. :green_circle: Upgrade to Rancher v2.9.0

    * upgrade rancher  * RKE2 cluster still running  * nginx still works  

    Ref. https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades

12. :green_circle: Upgrade to RKE2 v1.29

    * upgrade RKE2 version  * RKE2 cluster still running  * nginx still works  

[albinsun]

Upgrade to harvester-v1.3.1 to harvester-v1.3.2-rc2

1. :green_circle: Prepare an airgapped upgrade

   

   Ref. https://docs.harvesterhci.io/v1.3/upgrade/index/#prepare-an-air-gapped-upgrade

2. :green_circle: Trigger Upgrade

    
3. :green_circle: Successfully upgrade

   

[albinsun]

Post-upgrade Checks

1. Dependencies Check Chart: https://github.com/harvester/harvester/tree/v1.3/deploy/charts/harvester/dependency_charts Values: https://github.com/harvester/harvester/blob/v1.3/deploy/charts/harvester/values.yaml

   • :green_circle: csi-snapshotter

     
   • :green_circle: kubevirt-operator

     
   • :green_circle: kubevirt

     
   • :green_circle: snapshot-validation-webhook

     
   • :green_circle: whereabouts

     

2. Check dependency_charts version https://github.com/harvester/harvester/tree/v1.3/deploy/charts/harvester-crd/templates

   • :green_circle: csi-snapshotter

     * volumesnapshotclasses  * volumesnapshotcontents  * volumesnapshots 
   • :green_circle: kubevirt-operator

     
   • :green_circle: whereabouts

     * whereabouts.cni.cncf.io_ippools  * whereabouts.cni.cncf.io_overlappingrangeipreservations 

3. Check harvester_crd version https://github.com/harvester/harvester/tree/v1.3/deploy/charts/harvester-crd/templates

   • :green_circle: harvesterhci.io_addons.yaml

     
   • :green_circle: harvesterhci.io_keypairs.yaml

     
   • :green_circle: harvesterhci.io_preferences.yaml

     
   • :green_circle: harvesterhci.io_settings.yaml

     
   • :green_circle: harvesterhci.io_supportbundles.yaml

     
   • :green_circle: harvesterhci.io_upgradelogs.yaml

     
   • :green_circle: harvesterhci.io_upgrades.yaml

     
   • :green_circle: harvesterhci.io_versions.yaml

     
   • :green_circle: harvesterhci.io_virtualmachinebackups.yaml

     
   • :green_circle: harvesterhci.io_virtualmachineimages.yaml

     
   • :green_circle: harvesterhci.io_virtualmachinerestores.yaml

     
   • :green_circle: harvesterhci.io_virtualmachinetemplates.yaml

     
   • :green_circle: harvesterhci.io_virtualmachinetemplateversions.yaml

     
4. :green_circle: Virtual machines are in same state as before and accessible.

   * VMs  * sles15sp6  * ubuntu24  * rhel9: Off
5. :green_circle: Restore the backups, check the data

   Restore New * sles15sp6  * rhel9  Restore Replace * rhel9  
6. :green_circle: Image and volume status

   * Image  * Volume 
7. :green_circle: Monitoring chart status

   
8. :green_circle: VM operations are highlighted and working fine.

   - [x] Stop  - [x] Start  - [x] Restart - [x] Soft Reboot - [x] Migrate to witness node should fail  - [x] Migrate to default node - [x] Add Volume
9. :green_circle: RKE2 guest cluster still works

   * cluster  * LB 
10. :green_circle: Upgrade the RKE2 guest cluster to 1.30

    
11. :green_circle: Upgrade cloud provider and CSI driver

     * cloud-provider  * csi-driver  > cloud-provider >  > csi-driver > 
12. :green_circle: Verify DHCP load balancer service and create a new Harvester PVC

    * LB  * New PVC 
13. :green_circle: Shutting off VM and then restarting VM

    - [x] Stop  - [x] Start 
14. :green_circle: Add a (Compute) node after the upgrade