search

kubevirt / containerized-data-importer

Data Import Service for kubernetes, designed with kubevirt in mind.
Apache License 2.0
400 stars 254 forks source link

Attempting to Clone DataVolume #3122

Closed Smithx10 closed 3 months ago

Smithx10 commented 5 months ago

I am attempting to configure "block" storage profiles with openebs/zfs-localpv to deploy a vm using a fedora block pv.

I've been trying to find documentation on how to do this, and am coming up short. Additional eyes / Help would be much appreciated.

Where I get stuck

4s                       Normal    SuccessfulDataVolumeCreate       VirtualMachine/bruce          Created DataVolume bruce
3s                       Normal    CloneScheduled                   DataVolume/bruce              Cloning from vm-images/fedora into default/bruce scheduled
3s                       Warning   IncompatibleVolumeModes          PersistentVolumeClaim/bruce   The volume modes of source and target are incompatible
3s                       Warning   ProvisioningFailed               PersistentVolumeClaim/bruce   Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "bruce": the object has been modified; please apply your changes to the latest version and try again
3s                       Normal    CloneInProgress                  DataVolume/bruce              Cloning from vm-images/fedora into default/bruce in progress
3s                       Warning   Pending                          DataVolume/bruce              Clone Pending
3s                       Normal    Pending                          DataVolume/bruce              PVC bruce Pending

VM Defintion

root@rke2-s0:~# k get pvc -n vm-images fedora -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Import Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.populator.progress: 100.0%
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: zfs.csi.openebs.io
    volume.kubernetes.io/storage-provisioner: zfs.csi.openebs.io
  creationTimestamp: "2024-03-04T01:40:23Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
  name: fedora
  namespace: vm-images
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: fedora
    uid: 36cfe030-2bcd-4587-88de-cbe479abdc71
  resourceVersion: "58875"
  uid: 1090fe4d-69c8-43a4-a02c-c40e92441d55
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-36cfe030-2bcd-4587-88de-cbe479abdc71
  dataSourceRef:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-36cfe030-2bcd-4587-88de-cbe479abdc71
  resources:
    requests:
      storage: "5368709120"
  storageClassName: volumes
  volumeMode: Block
  volumeName: pvc-55cff474-9f22-4dde-b980-46cebf8b7fde
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  phase: Bound
root@rke2-s0:~# ^C
root@rke2-s0:~# ^C
root@rke2-s0:~#
root@rke2-s0:~# cat vm1_pvc.yml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: bruce
  name: bruce
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: bruce
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumedisk1
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 1Gi
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: bruce
        name: datavolumedisk1
      - cloudInitNoCloud:
          userData: |
            #cloud-config
            hostname: vm1
            ssh_pwauth: True
            disable_root: false
            ssh_authorized_keys:
            - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/
        name: cloudinitdisk
  dataVolumeTemplates:
  - metadata:
      name: bruce
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
      source:
        pvc:
          namespace: vm-images
          name: fedora

Fedora PVC

root@rke2-s0:~# k get pvc -n vm-images fedora -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Import Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.populator.progress: 100.0%
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: zfs.csi.openebs.io
    volume.kubernetes.io/storage-provisioner: zfs.csi.openebs.io
  creationTimestamp: "2024-03-04T01:40:23Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
  name: fedora
  namespace: vm-images
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: fedora
    uid: 36cfe030-2bcd-4587-88de-cbe479abdc71
  resourceVersion: "58875"
  uid: 1090fe4d-69c8-43a4-a02c-c40e92441d55
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-36cfe030-2bcd-4587-88de-cbe479abdc71
  dataSourceRef:
    apiGroup: cdi.kubevirt.io
    kind: VolumeImportSource
    name: volume-import-source-36cfe030-2bcd-4587-88de-cbe479abdc71
  resources:
    requests:
      storage: "5368709120"
  storageClassName: volumes
  volumeMode: Block
  volumeName: pvc-55cff474-9f22-4dde-b980-46cebf8b7fde
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  phase: Bound

Datavolume source

root@rke2-s0:~# cat dv_fedora.yml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "fedora"
  namespace: "vm-images"
spec:
  storage:
    resources:
      requests:
        storage: 5Gi
  source:
    http:
      url: "http://192.168.122.1:8080/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"

Snap Class

root@rke2-s0:~# k get volumesnapshotclass zfspv-snapclass -o yaml
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: zfs.csi.openebs.io
kind: VolumeSnapshotClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"snapshot.storage.k8s.io/v1","deletionPolicy":"Delete","driver":"zfs.csi.openebs.io","kind":"VolumeSnapshotClass","metadata":{"annotations":{"snapshot.storage.kubernetes.io/is-default-class":"true"},"name":"zfspv-snapclass"}}
    snapshot.storage.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-03-04T00:48:03Z"
  generation: 1
  name: zfspv-snapclass
  resourceVersion: "29181"
  uid: 64693609-02a9-4499-b65d-65e3139e88f5

Storage Class

root@rke2-s0:~# k get sc volumes -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"volumes"},"parameters":{"poolname":"system/volumes"},"provisioner":"zfs.csi.openebs.io"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-03-04T00:15:51Z"
  name: volumes
  resourceVersion: "56911"
  uid: 62d16081-82b9-4a29-b67e-9c37e3dc719d
parameters:
  poolname: system/volumes
provisioner: zfs.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Storage Profile

root@rke2-s0:~# k get storageprofile volumes -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  creationTimestamp: "2024-03-04T00:15:51Z"
  generation: 6
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    cdi.kubevirt.io: ""
  name: volumes
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 678c50ed-3e73-4c53-9bef-13b37537c10f
  resourceVersion: "29183"
  uid: d08facfe-609f-42c9-a538-1a29ac405cd1
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    volumeMode: Block
  cloneStrategy: csi-clone
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    volumeMode: Block
  cloneStrategy: csi-clone
  dataImportCronSourceFormat: pvc
  provisioner: zfs.csi.openebs.io
  snapshotClass: zfspv-snapclass
  storageClass: volumes

Kube version

root@rke2-s0:~# kubectl version
Client Version: v1.29.0+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0+rke2r1

Kernel

Linux rke2-s0 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

OS Release

root@rke2-s0:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
BDFV2_RELEASE="20240302T005108Z"

CDI Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    operator.cdi.kubevirt.io/lastAppliedConfiguration: '{"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"cdi-deployment","namespace":"cdi","creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"containerized-data-importer"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"volumes":[{"name":"cdi-api-signing-key","secret":{"secretName":"cdi-api-signing-key","items":[{"key":"id_rsa.pub","path":"id_rsa.pub"},{"key":"id_rsa","path":"id_rsa"}],"defaultMode":420}},{"name":"uploadserver-ca-cert","secret":{"secretName":"cdi-uploadserver-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-client-ca-cert","secret":{"secretName":"cdi-uploadserver-client-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-ca-bundle","configMap":{"name":"cdi-uploadserver-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}},{"name":"uploadserver-client-ca-bundle","configMap":{"name":"cdi-uploadserver-client-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}}],"containers":[{"name":"cdi-controller","image":"quay.io/kubevirt/cdi-controller:v1.58.1","args":["-v=1"],"ports":[{"name":"metrics","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"IMPORTER_IMAGE","value":"quay.io/kubevirt/cdi-importer:v1.58.1"},{"name":"CLONER_IMAGE","value":"quay.io/kubevirt/cdi-cloner:v1.58.1"},{"name":"UPLOADSERVER_IMAGE","value":"quay.io/kubevirt/cdi-uploadserver:v1.58.1"},{"name":"UPLOADPROXY_SERVICE","value":"cdi-uploadproxy"},{"name":"PULL_POLICY","value":"IfNotPresent"},{"name":"INSTALLER_PART_OF_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/part-of'']"}}},{"name":"INSTALLER_VERSION_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/version'']"}}}],"resources":{"requests":{"cpu":"100m","memory":"150Mi"}},"volumeMounts":[{"name":"cdi-api-signing-key","mountPath":"/var/run/cdi/token/keys"},{"name":"uploadserver-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-signer"},{"name":"uploadserver-client-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-client-signer"},{"name":"uploadserver-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-signer-bundle"},{"name":"uploadserver-client-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-client-signer-bundle"}],"readinessProbe":{"exec":{"command":["cat","/tmp/ready"]},"initialDelaySeconds":2,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["ALL"]},"runAsNonRoot":true,"allowPrivilegeEscalation":false,"seccompProfile":{"type":"RuntimeDefault"}}}],"nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"cdi-sa","securityContext":{"runAsNonRoot":true},"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"priorityClassName":"kubevirt-cluster-critical"}},"strategy":{}},"status":{}}'
  creationTimestamp: "2024-03-04T00:04:11Z"
  generation: 1
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-operator
    cdi.kubevirt.io: ""
    operator.cdi.kubevirt.io/createVersion: v1.58.1
    prometheus.cdi.kubevirt.io: "true"
  name: cdi-deployment
  namespace: cdi
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 678c50ed-3e73-4c53-9bef-13b37537c10f
  resourceVersion: "51109"
  uid: 2a4bdd31-1707-4615-9728-b445610f3b0d
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: containerized-data-importer
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-operator
        cdi.kubevirt.io: ""
        operator.cdi.kubevirt.io/createVersion: v1.58.1
        prometheus.cdi.kubevirt.io: "true"
    spec:
      containers:
      - args:
        - -v=1
        env:
        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.58.1
        - name: CLONER_IMAGE
          value: quay.io/kubevirt/cdi-cloner:v1.58.1
        - name: UPLOADSERVER_IMAGE
          value: quay.io/kubevirt/cdi-uploadserver:v1.58.1
        - name: UPLOADPROXY_SERVICE
          value: cdi-uploadproxy
        - name: PULL_POLICY
          value: IfNotPresent
        - name: INSTALLER_PART_OF_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/part-of']
        - name: INSTALLER_VERSION_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/version']
        image: quay.io/kubevirt/cdi-controller:v1.58.1
        imagePullPolicy: IfNotPresent
        name: cdi-controller
        ports:
        - containerPort: 8080
          name: metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
          failureThreshold: 3
          initialDelaySeconds: 2
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 150Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/run/cdi/token/keys
          name: cdi-api-signing-key
        - mountPath: /var/run/certs/cdi-uploadserver-signer
          name: uploadserver-ca-cert
        - mountPath: /var/run/certs/cdi-uploadserver-client-signer
          name: uploadserver-client-ca-cert
        - mountPath: /var/run/ca-bundle/cdi-uploadserver-signer-bundle
          name: uploadserver-ca-bundle
        - mountPath: /var/run/ca-bundle/cdi-uploadserver-client-signer-bundle
          name: uploadserver-client-ca-bundle
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: kubevirt-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
      serviceAccount: cdi-sa
      serviceAccountName: cdi-sa
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - name: cdi-api-signing-key
        secret:
          defaultMode: 420
          items:
          - key: id_rsa.pub
            path: id_rsa.pub
          - key: id_rsa
            path: id_rsa
          secretName: cdi-api-signing-key
      - name: uploadserver-ca-cert
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          secretName: cdi-uploadserver-signer
      - name: uploadserver-client-ca-cert
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          secretName: cdi-uploadserver-client-signer
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-signer-bundle
        name: uploadserver-ca-bundle
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-client-signer-bundle
        name: uploadserver-client-ca-bundle
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2024-03-04T00:04:11Z"
    lastUpdateTime: "2024-03-04T00:04:24Z"
    message: ReplicaSet "cdi-deployment-74b786dcc6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-03-04T01:28:56Z"
    lastUpdateTime: "2024-03-04T01:28:56Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
akalenyu commented 5 months ago

dataVolumeTemplates:

  • metadata: name: bruce spec: pvc: accessModes:
    • ReadWriteOnce resources: requests: storage: 5Gi

I think the issue is here, if you want to infer from the StorageProfile you created, you need to use storage instead of pvc: https://github.com/kubevirt/containerized-data-importer/blob/main/doc/storageprofile.md#handling-the-dv-with-defaults-from-storage-profiles

I think what happens is that you are using a raw block ZFS storage class, but this is a Filesystem PVC request.

awels commented 5 months ago

Yes I think Alex is correct, if omited the volumeMode defaults to filesystem, and the source is block, and thus doing a clone will fail. By using the pvc stanza instead of storage CDI will use whatever you pass it and not infer anything from the storage profile. I am assuming chaning it fixed it since this is 2 weeks old. Can we close this issue?

aglitke commented 3 months ago

I am going to close this issue due to inactivity. Please feel free to reopen if you'd like to continue the investigation.