Failed to Clone VM When DataVolume's Type is WaitForFirstConsumer

SSmallMonster commented 9 months ago

What happened: Cloning from one VM to another fails when using a data volume of type WaitForFirstConsumer as the VM's data volume.

What you expected to happen: Virtual machines can be successfully cloned.

How to reproduce it (as minimally and precisely as possible):

Create VirtualMachine Using DataVolume of type WaitForFirstConsumer

---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
kubevirt.io/vm: vm-alpine-datavolume
name: vm-alpine-datavolume
spec:
dataVolumeTemplates:
- metadata:
  creationTimestamp: null
  name: alpine-dv
spec:
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
    storageClassName: hwameistor-storage-lvm-hdd
  source:
    registry:
      url: docker://docker.io/kubevirt/alpine-container-disk-demo:devel
running: false
template:
metadata:
  labels:
    kubevirt.io/vm: vm-alpine-datavolume
spec:
  domain:
    devices:
      disks:
      - disk:
          bus: virtio
        name: datavolumedisk1
    resources:
      requests:
        memory: 128Mi
  terminationGracePeriodSeconds: 0
  volumes:
  - dataVolume:
      name: alpine-dv
    name: datavolumedisk1

Create VirtualMachineClone

kind: VirtualMachineClone
apiVersion: "clone.kubevirt.io/v1alpha1"
metadata:
name: testclone
spec:
# source & target definitions
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: vm-alpine-datavolume
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: vm-clone-vm-alpine-datavolume

Additional context: The CSI Provisioner I'm using is local storage, which has the VOLUMEBINDINGMODE mode WaitForFirstConsumer.

Here's some information on some resources that will hopefully help:

vm info:

[root@172-30-46-10 ~]# kubectl get vm
NAME                            AGE     STATUS                    READY
vm-alpine-datavolume            3h13m   Stopped                   False
vm-clone-vm-alpine-datavolume   21m     WaitingForVolumeBinding   False

pvc info:

[root@172-30-46-10 ~]# kubectl get pvc
NAME                                                           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                 AGE
alpine-dv                                                      Bound     pvc-763e001c-40a8-47b9-b04e-ae3252a9df59   2Gi        RWO            hwameistor-storage-lvm-hdd   3h13m
restore-5fa71e75-5075-4eda-8413-8b5a1267da16-datavolumedisk1   Pending                                                                        hwameistor-storage-lvm-hdd   21m
[root@172-30-46-10 ~]# kubectl describe pvc restore-5fa71e75-5075-4eda-8413-8b5a1267da16-datavolumedisk1
Name:          restore-5fa71e75-5075-4eda-8413-8b5a1267da16-datavolumedisk1
Namespace:     default
StorageClass:  hwameistor-storage-lvm-hdd
Status:        Pending
Volume:
...
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      vmsnapshot-f07a787b-b084-48cc-b59f-c1231bf254d1-volume-datavolumedisk1
Used By:     virt-launcher-vm-clone-vm-alpine-datavolume-drhgm
Events:
  Type     Reason                Age                   From                                                                                                             Message
  ----     ------                ----                  ----                                                                                                             -------
  Normal   WaitForFirstConsumer  21m (x5 over 21m)     persistentvolume-controller                                                                                      waiting for first consumer to be created before binding
  Normal   Provisioning          2m37s (x13 over 21m)  lvm.hwameistor.io_hwameistor-local-storage-csi-controller-665f64bc4d-nxs85_f12404b0-8b37-41a5-8004-e6b55690f794  External provisioner is provisioning volume for claim "default/restore-5fa71e75-5075-4eda-8413-8b5a1267da16-datavolumedisk1"
  Warning  ProvisioningFailed    2m37s (x13 over 21m)  lvm.hwameistor.io_hwameistor-local-storage-csi-controller-665f64bc4d-nxs85_f12404b0-8b37-41a5-8004-e6b55690f794  failed to provision volume with StorageClass "hwameistor-storage-lvm-hdd": error getting handle for DataSource Type VolumeSnapshot by Name vmsnapshot-f07a787b-b084-48cc-b59f-c1231bf254d1-volume-datavolumedisk1: snapshot vmsnapshot-f07a787b-b084-48cc-b59f-c1231bf254d1-volume-datavolumedisk1 is currently being deleted

storageclass info:

[root@172-30-46-10 ~]# kubectl get sc
NAME                                   PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
hwameistor-storage-lvm-hdd (default)   lvm.hwameistor.io            Delete          WaitForFirstConsumer   true                   5d3h

vmclone info:

[root@172-30-46-10 ~]# kubectl get vmclone
NAME        PHASE       SOURCEVIRTUALMACHINE   TARGETVIRTUALMACHINE
testclone   Succeeded   vm-alpine-datavolume   vm-clone-vm-alpine-datavolume

vmrestore info:

[root@172-30-46-10 ~]# kubectl get vmrestore -w
NAME                                                 TARGETKIND       TARGETNAME                      COMPLETE   RESTORETIME   ERROR
clone-testclone-restore-vm-alpine-datavolume-6lhfw   VirtualMachine   vm-clone-vm-alpine-datavolume
clone-testclone-restore-vm-alpine-datavolume-6lhfw   VirtualMachine   vm-clone-vm-alpine-datavolume   false
clone-testclone-restore-vm-alpine-datavolume-6lhfw   VirtualMachine   vm-clone-vm-alpine-datavolume   true       0s
clone-testclone-restore-vm-alpine-datavolume-8rlx2   VirtualMachine   vm-clone-vm-alpine-datavolume   true       0s

Environment:

KubeVirt version (use virtctl version): N/A

Client Version: version.Info{GitVersion:"v0.52.0", GitCommit:"836afc8c556206fc7715615ddb6516d76f4240db", GitTreeState:"clean", BuildDate:"2022-04-08T16:25:37Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v1.0.0", GitCommit:"579bb66545630d93b29d0399219aaa51c3f0a00b", GitTreeState:"clean", BuildDate:"2023-07-06T18:46:13Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes version (use kubectl version): N/A

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

VM or VMI specifications: N/A
Cloud provider or hardware configuration: N/A

OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

Kernel (e.g. uname -a):

Linux 172-30-46-10 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Install tools: N/A
Others: N/A

SSmallMonster commented 9 months ago

I think the snapshot being deleted should be the main reason for the failure, plus VMRestore gets marked successful very quickly after creation.The reason for the successful mark is that the target vm is not created, which always holds true in my environment.

https://github.com/kubevirt/kubevirt/blob/9ad24545b688aeb95b3aa292e6516273cc808ff8/pkg/storage/snapshot/restore.go#L374-L377

I hope this information helps, let me know if you need anything else!

akalenyu commented 9 months ago

Thanks for submitting and digging through this!

I think the snapshot being deleted should be the main reason for the failure

Yep I think you're right, @fossedihelm was seeing the same in https://bugzilla.redhat.com/show_bug.cgi?id=2237418

fossedihelm commented 9 months ago

@akalenyu @SSmallMonster Correct! I am working on a patch for it :) Thanks /assign

SSmallMonster commented 9 months ago

@akalenyu Thank you for your kind reply.

On another note, I noticed that StorageProfile has a cloneStrategy configuration, which I set to csi-clone, but it doesn't look like it's going through CSI Provisioner to create a clone data volume. I'm not sure if I'm missing something or misinterpreting it.

SSmallMonster commented 9 months ago

@akalenyu @SSmallMonster Correct! I am working on a patch for it :) Thanks /assign

That's great! Thanks!!

akalenyu commented 9 months ago

@akalenyu Thank you for your kind reply.

On another note, I noticed that StorageProfile has a cloneStrategy configuration, which I set to csi-clone, but it doesn't look like it's going through CSI Provisioner to create a clone data volume. I'm not sure if I'm missing something or misinterpreting it.

So VMClone (or other kubevirt APIs) are unaware of the StorageProfile API. This is only taken into account in context of import/upload/clone (CDI operations). Behind the curtains, VMClone relies on VMSnapshot/VMRestore which rely on VolumeSnapshots so csi clone will likely also not achieve the desired result

SSmallMonster commented 9 months ago

@akalenyu Thank you for your kind reply.

On another note, I noticed that StorageProfile has a cloneStrategy configuration, which I set to csi-clone, but it doesn't look like it's going through CSI Provisioner to create a clone data volume. I'm not sure if I'm missing something or misinterpreting it.

So VMClone (or other kubevirt APIs) are unaware of the StorageProfile API. This is only taken into account in context of import/upload/clone (CDI operations).

Behind the curtains, VMClone relies on VMSnapshot/VMRestore which rely on VolumeSnapshots so csi clone will likely also not achieve the desired result

Got it.Thanks!

kubevirt / kubevirt

Failed to Clone VM When DataVolume's Type is WaitForFirstConsumer #10832