Error mounting volume when creating a new vm

[KyleSponz]

Hello all I just created my first cluster and am getting the following error when starting a new VM

AttachVolume.Attach failed for volume "pvc-5c7440eb-e1d3-4ad9-ac64-1ae3a569057c" : rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [message=unable to attach volume pvc-5c7440eb-e1d3-4ad9-ac64-1ae3a569057c to test3: invalid volume state to start migration attaching, code=Server Error, detail=] from [http://longhorn-backend:9500/v1/volumes/pvc-5c7440eb-e1d3-4ad9-ac64-1ae3a569057c?action=attach]

The volume shows up in volumes tab this is my first time using Harvester so please tell me what else you need and where to find it and apologies if I missed anything

[w13915984028]

@KyleSponz Could you please add more details, thanks.

The Harvester version The cluster scale, single node or multi node, node number. The node resource, CPU cores, memory, storage Could you also attach a few pictures or logs about that status. And kubectl get pvc -A -o YAML, kubectl get volume -A -o YAML, kubectl get vm -A -o YAML ...

It is possble that the VM's claimed storage size is big than the usable space, that causes the PVC is unable to be attached.

[KyleSponz]

Hi @w13915984028 sorry for the delay, we had a hardware failure that took me away from this part The cluster is 3 identical nodes 64 core 252 gb ram 12tb each of storage dual 10gb nics

---

kubectl get pvc -A -o YAML apiVersion: v1 items:

• apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-24T22:07:02Z" finalizers:
  • kubernetes.io/pvc-protection labels: alertmanager: rancher-monitoring-alertmanager app: alertmanager app.kubernetes.io/instance: rancher-monitoring-alertmanager app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: alertmanager name: alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0 namespace: cattle-monitoring-system resourceVersion: "7015" uid: c21d3308-2264-417c-8996-07d8ca7bf304 spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 5Gi storageClassName: harvester-longhorn volumeMode: Filesystem volumeName: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 status: accessModes:
  • ReadWriteOnce capacity: storage: 5Gi phase: Bound
• apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-24T22:07:07Z" finalizers:
  • kubernetes.io/pvc-protection labels: app: prometheus app.kubernetes.io/instance: rancher-monitoring-prometheus app.kubernetes.io/managed-by: prometheus-operator app.kubernetes.io/name: prometheus operator.prometheus.io/name: rancher-monitoring-prometheus operator.prometheus.io/shard: "0" prometheus: rancher-monitoring-prometheus name: prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0 namespace: cattle-monitoring-system resourceVersion: "7010" uid: 89bf3461-36db-417b-a645-e0055e66a7a1 spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 50Gi storageClassName: harvester-longhorn volumeMode: Filesystem volumeName: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 status: accessModes:
  • ReadWriteOnce capacity: storage: 50Gi phase: Bound
• apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: meta.helm.sh/release-name: rancher-monitoring meta.helm.sh/release-namespace: cattle-monitoring-system objectset.rio.cattle.io/id: default-mcc-rancher-monitoring-cattle-fleet-local-system pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-24T22:06:55Z" finalizers:
  • kubernetes.io/pvc-protection labels: app.kubernetes.io/instance: rancher-monitoring app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: grafana app.kubernetes.io/version: 7.5.11 helm.sh/chart: grafana-6.16.14 objectset.rio.cattle.io/hash: a8c87f2d01731fdad3b988f675a7c8a7da10d382 name: rancher-monitoring-grafana namespace: cattle-monitoring-system resourceVersion: "7019" uid: 9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 spec: accessModes:
  • ReadWriteOnce resources: requests: storage: 2Gi storageClassName: harvester-longhorn volumeMode: Filesystem volumeName: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 status: accessModes:
  • ReadWriteOnce capacity: storage: 2Gi phase: Bound
• apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: harvesterhci.io/imageId: default/image-6plqj harvesterhci.io/owned-by: '[{"schema":"kubevirt.io.virtualmachine","refs":["default/testubnt"]}]' pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io volume.kubernetes.io/storage-provisioner: driver.longhorn.io creationTimestamp: "2022-10-25T22:27:01Z" finalizers:
  • kubernetes.io/pvc-protection name: testubnt-disk-0-9ij4j namespace: default resourceVersion: "1347180" uid: d1406372-d38b-4b2e-ae61-fbd861d30437 spec: accessModes:
  • ReadWriteMany resources: requests: storage: 10Gi storageClassName: longhorn-image-6plqj volumeMode: Block volumeName: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 status: accessModes:
  • ReadWriteMany capacity: storage: 10Gi phase: Bound kind: List metadata: resourceVersion: ""

    ---

kubectl get volume -A -o YAML apiVersion: v1 items:

• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 recurring-job-group.longhorn.io/default: enabled name: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 namespace: longhorn-system resourceVersion: "2651573" uid: 6757581d-417b-4c69-824f-5344a09e0ca3 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "53687091200" staleReplicaTimeout: 30 status: actualSize: 7673200640 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:26Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:26Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 pvStatus: Bound pvcName: prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0 workloadsStatus:
    • podName: prometheus-rancher-monitoring-prometheus-0 podStatus: Running workloadName: prometheus-rancher-monitoring-prometheus workloadType: StatefulSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 recurring-job-group.longhorn.io/default: enabled name: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 namespace: longhorn-system resourceVersion: "2651229" uid: a5bf5183-7b7c-4f2d-94f9-42a467893786 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "2147483648" staleReplicaTimeout: 30 status: actualSize: 135438336 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 pvStatus: Bound pvcName: rancher-monitoring-grafana workloadsStatus:
    • podName: rancher-monitoring-grafana-8559b68898-sczhm podStatus: Running workloadName: rancher-monitoring-grafana-8559b68898 workloadType: ReplicaSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 recurring-job-group.longhorn.io/default: enabled name: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 namespace: longhorn-system resourceVersion: "2646567" uid: bbdd64e4-0c56-45a7-84c0-1eb3f4920b31 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "5368709120" staleReplicaTimeout: 30 status: actualSize: 164610048 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 pvStatus: Bound pvcName: alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0 workloadsStatus:
    • podName: alertmanager-rancher-monitoring-alertmanager-0 podStatus: Running workloadName: alertmanager-rancher-monitoring-alertmanager workloadType: StatefulSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-25T22:27:07Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 recurring-job-group.longhorn.io/default: enabled name: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 namespace: longhorn-system resourceVersion: "1347282" uid: 093571e5-0a51-4864-95c4-d80e2b6c1b00 spec: Standby: false accessMode: rwx backingImage: default-image-6plqj baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: true migrationNodeID: "" nodeID: csp3 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "10737418240" staleReplicaTimeout: 30 status: actualSize: 0 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp3 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: default pvName: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 pvStatus: Bound pvcName: testubnt-disk-0-9ij4j workloadsStatus:
    • podName: virt-launcher-testubnt-2j6p9 podStatus: Pending workloadName: testubnt workloadType: VirtualMachineInstance lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp3 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: unknown shareEndpoint: "" shareState: "" state: attaching kind: List metadata: resourceVersion: ""

      ---

kubectl get volume -A -o YAML apiVersion: v1 items:

• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 recurring-job-group.longhorn.io/default: enabled name: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 namespace: longhorn-system resourceVersion: "2651573" uid: 6757581d-417b-4c69-824f-5344a09e0ca3 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "53687091200" staleReplicaTimeout: 30 status: actualSize: 7673200640 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:26Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:26Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-89bf3461-36db-417b-a645-e0055e66a7a1 pvStatus: Bound pvcName: prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0 workloadsStatus:
    • podName: prometheus-rancher-monitoring-prometheus-0 podStatus: Running workloadName: prometheus-rancher-monitoring-prometheus workloadType: StatefulSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 recurring-job-group.longhorn.io/default: enabled name: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 namespace: longhorn-system resourceVersion: "2651229" uid: a5bf5183-7b7c-4f2d-94f9-42a467893786 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "2147483648" staleReplicaTimeout: 30 status: actualSize: 135438336 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-9b0b45a1-5c6b-4fdf-8c53-bf8497925a88 pvStatus: Bound pvcName: rancher-monitoring-grafana workloadsStatus:
    • podName: rancher-monitoring-grafana-8559b68898-sczhm podStatus: Running workloadName: rancher-monitoring-grafana-8559b68898 workloadType: ReplicaSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-24T22:07:25Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 recurring-job-group.longhorn.io/default: enabled name: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 namespace: longhorn-system resourceVersion: "2646567" uid: bbdd64e4-0c56-45a7-84c0-1eb3f4920b31 spec: Standby: false accessMode: rwo backingImage: "" baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: false migrationNodeID: "" nodeID: csp1 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "5368709120" staleReplicaTimeout: 30 status: actualSize: 164610048 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:54:22Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-24T22:07:25Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp1 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: cattle-monitoring-system pvName: pvc-c21d3308-2264-417c-8996-07d8ca7bf304 pvStatus: Bound pvcName: alertmanager-rancher-monitoring-alertmanager-db-alertmanager-rancher-monitoring-alertmanager-0 workloadsStatus:
    • podName: alertmanager-rancher-monitoring-alertmanager-0 podStatus: Running workloadName: alertmanager-rancher-monitoring-alertmanager workloadType: StatefulSet lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp1 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: healthy shareEndpoint: "" shareState: "" state: attached
• apiVersion: longhorn.io/v1beta2 kind: Volume metadata: creationTimestamp: "2022-10-25T22:27:07Z" finalizers:
  • longhorn.io generation: 2 labels: longhornvolume: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 recurring-job-group.longhorn.io/default: enabled name: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 namespace: longhorn-system resourceVersion: "1347282" uid: 093571e5-0a51-4864-95c4-d80e2b6c1b00 spec: Standby: false accessMode: rwx backingImage: default-image-6plqj baseImage: "" dataLocality: disabled dataSource: "" disableFrontend: false diskSelector: [] encrypted: false engineImage: longhornio/longhorn-engine:v1.3.1 fromBackup: "" frontend: blockdev lastAttachedBy: "" migratable: true migrationNodeID: "" nodeID: csp3 nodeSelector: [] numberOfReplicas: 3 recurringJobs: [] replicaAutoBalance: ignored revisionCounterDisabled: false size: "10737418240" staleReplicaTimeout: 30 status: actualSize: 0 cloneStatus: snapshot: "" sourceVolume: "" state: "" conditions:
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "False" type: toomanysnapshots
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "True" type: scheduled
  • lastProbeTime: "" lastTransitionTime: "2022-10-25T22:27:08Z" message: "" reason: "" status: "False" type: restore currentImage: longhornio/longhorn-engine:v1.3.1 currentNodeID: csp3 expansionRequired: false frontendDisabled: false isStandby: false kubernetesStatus: lastPVCRefAt: "" lastPodRefAt: "" namespace: default pvName: pvc-d1406372-d38b-4b2e-ae61-fbd861d30437 pvStatus: Bound pvcName: testubnt-disk-0-9ij4j workloadsStatus:
    • podName: virt-launcher-testubnt-2j6p9 podStatus: Pending workloadName: testubnt workloadType: VirtualMachineInstance lastBackup: "" lastBackupAt: "" lastDegradedAt: "" ownerID: csp3 pendingNodeID: "" remountRequestedAt: "" restoreInitiated: false restoreRequired: false robustness: unknown shareEndpoint: "" shareState: "" state: attaching kind: List metadata: resourceVersion: "" csp3:/home/rancher # kubectl get vm -A -o YAML apiVersion: v1 items:
• apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: harvesterhci.io/vmRunStrategy: RerunOnFailure harvesterhci.io/volumeClaimTemplates: '[{"metadata":{"name":"testubnt-disk-0-9ij4j","annotations":{"harvesterhci.io/imageId":"default/image-6plqj"}},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"10Gi"}},"volumeMode":"Block","storageClassName":"longhorn-image-6plqj"}}]' kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1alpha3 network.harvesterhci.io/ips: '[]' creationTimestamp: "2022-10-25T22:27:01Z" finalizers:
  • wrangler.cattle.io/VMController.UnsetOwnerOfPVCs generation: 1 labels: harvesterhci.io/creator: harvester harvesterhci.io/os: ubuntu name: testubnt namespace: default resourceVersion: "1347256" uid: 297e762d-5dc7-4f16-ab89-9217ac7a126d spec: runStrategy: RerunOnFailure template: metadata: annotations: harvesterhci.io/sshNames: '[]' creationTimestamp: null labels: harvesterhci.io/vmName: testubnt spec: domain: cpu: cores: 2 sockets: 1 threads: 1 devices: disks:
    • bootOrder: 1 cdrom: bus: sata name: disk-0
    • disk: bus: virtio name: cloudinitdisk inputs:
    • bus: usb name: tablet type: tablet interfaces:
    • masquerade: {} model: virtio name: default features: acpi: enabled: true machine: type: q35 memory: guest: 8092Mi resources: limits: cpu: "2" memory: 8Gi requests: cpu: 125m memory: 5461Mi evictionStrategy: LiveMigrate hostname: testubnt networks:
      • name: default pod: {} volumes:
      • name: disk-0 persistentVolumeClaim: claimName: testubnt-disk-0-9ij4j
      • cloudInitNoCloud: networkDataSecretRef: name: testubnt-sfaq2 secretRef: name: testubnt-sfaq2 name: cloudinitdisk status: conditions:
  • lastProbeTime: "2022-10-25T22:27:02Z" lastTransitionTime: "2022-10-25T22:27:02Z" message: Guest VM is not reported as running reason: GuestNotRunning status: "False" type: Ready created: true printableStatus: Starting volumeSnapshotStatuses:
  • enabled: false name: disk-0 reason: 2 matching VolumeSnapshotClasses for longhorn-image-6plqj
  • enabled: false name: cloudinitdisk reason: Snapshot is not supported for this volumeSource type [cloudinitdisk] kind: List metadata: resourceVersion: ""

[KyleSponz]

It turns out it was a corrupted ISO, I was using the web URL download function, direct upload of a known good file remediated the issue