[e2e] [BUG] Block device volume failed to unmount when it is detached unexpectedly

[harvesterhci-io-github-bot]

What's the test to develop? Please describe

Case 1: VM is not stuck in terminating

1. Create a 3-node cluster.
2. Check DaemonSet longhorn/longhorn-loop-device-cleaner is existent.
3. Create a VM.
4. Add a file test.txt with content 123 in the VM.
5. Check VM on which node and delete instance-manager-e-xxxx on that node.
6. Wait a minute, VM pod will go into terminating.
7. Wait a minute, a new VM pod will be created.
8. Check whether the content of the file test.txt is 123 in the VM.

As the longhorn-loop-device-cleaner is a workaround for the bug, we don't need to test Case 2 in the future where the workaround still exist.

Case 2: longhorn-loop-device-cleaner will be deleted after upgrade

1. Clone https://github.com/harvester/harvester
2. Remove longhorn-device-cleaner.yaml file under deploy/charts/harvester/template folder.
3. Build ISO.

   cp -r ~/.docker .
   export REPO=<dockerhub-username>
   export PUSH=true
   make
   make build-iso

4. Apply a new Version like:

   apiVersion: harvesterhci.io/v1beta1
   kind: Version
   metadata:
   name: new-master-head
   namespace: harvester-system
   spec:
   isoChecksum: <xxxx>
   isoURL: <localhost:port/harvester-master-amd64.iso>
   minUpgradableVersion: master-head
   releaseDate: "20220415"
   tags:
   - dev
   - test

5. Click the upgrade button on the dashboard.
6. After upgrade, longhorn/longhorn-loop-device-cleaner DaemonSet will be removed.

related issue: harvester/harvester#2137

[lanfon72]

This is a workaround solution, so it would be priority/3 to implement automation. (Or we will not need to implement it when the workaround is not needed.)