[TEST] Automate manual test cases in robot

[yangchiu]

What's the test to develop? Please describe

This ticket lists the manual test cases we've usually run during release testing.

When adding new test cases in robot, consider the following test cases first, so we can save our time from time consuming manual testing when doing release. cc @longhorn/qa @innobead

Describe the tasks for the test

• [ ] Normal Manual Test cases
  • [x] Replica Rebuilding
    • [x] https://github.com/longhorn/longhorn/issues/8731
  • [ ] Node drain and deletion test
    • [ ] https://github.com/longhorn/longhorn/issues/9292
  • [x] Single replica node down
    • https://github.com/longhorn/longhorn/issues/8350
  • [ ] Test Longhorn components recovery
  • [ ] Test backup listing S3/NFS
    • https://github.com/longhorn/longhorn/issues/8355
  • [ ] test backup functionality on google cloud and other s3 interop providers
  • [ ] pull backup created by another Longhorn system
  • [ ] The node the DR volume attached to is rebooted
    • https://github.com/longhorn/longhorn/issues/8425
  • [ ] Checksum enabled large volume with multiple rebuilding
  • [ ] HA Volume Migration
  • [ ] Test kubelet restart on a node of the cluster
  • [x] Node disconnection test
    • [x] https://github.com/longhorn/longhorn/issues/8617
  • [ ] The node the restore volume attached to is down
  • [ ] Physical node down
  • [x] Physical node reboot
    • https://github.com/longhorn/longhorn/issues/8567
• [ ] Uninstallation
  • [ ] Uninstallation Checks
    • [ ] https://github.com/longhorn/longhorn/issues/9222
• [ ] Upgrade
  • [ ] Kubernetes upgrade test
  • [ ] Longhorn Upgrade test
  • [ ] Test System Upgrade with New Instance Manager
  • [ ] Test system upgrade with a new storage class being default
  • [ ] Upgrade Conflict Handling test
  • [x] Test Engine Crash During Live Upgrade
    • https://github.com/longhorn/longhorn/issues/7859
  • [ ] DR volume live upgrade and rebuild
  • [ ] Test Backing Image during Longhorn upgrade
• [ ] CIS harden
  • [ ] Longhorn in an hardened cluster
  • [ ] Test Longhorn deployment on RKE2 v1.24- with CIS-1.6 profile
  • [ ] Test Longhorn deployment on RKE2 v1.25+ with CIS-1.23 profile
• [ ] Rancher related
  • [ ] Drain using Rancher UI
  • [ ] Upgrade Kubernetes using Rancher UI
• [ ] Volume Encryption
  • https://github.com/longhorn/longhorn/issues/7097

Additional context

[innobead]

@roger-ryao will drive this with @chriscchien . @yangchiu will continue focusing on negative testing instead.