search

kubevirt / containerized-data-importer

Data Import Service for kubernetes, designed with kubevirt in mind.
Apache License 2.0
400 stars 254 forks source link

cdi fails to clone filesystem #3119

Open Smithx10 opened 5 months ago

Smithx10 commented 5 months ago

Attempting to import and use Fedora from Example:

I import Fedora 

root@rke2-s0:~# kubectl logs -f importer-prime-67234c22-9b18-40be-aaa2-c8ad1d5f6744 -n vm-images
I0303 00:14:56.214817       1 importer.go:103] Starting importer
I0303 00:14:56.215021       1 importer.go:176] begin import process
I0303 00:14:57.200265       1 data-processor.go:356] Calculating available size
I0303 00:14:57.200331       1 data-processor.go:368] Checking out file system volume size.
I0303 00:14:57.200341       1 data-processor.go:376] Request image size not empty.
I0303 00:14:57.200370       1 data-processor.go:381] Target size 5681173672.
I0303 00:14:57.255964       1 data-processor.go:255] New phase: TransferDataFile
I0303 00:14:57.256315       1 util.go:194] Writing data...
I0303 00:14:58.256825       1 prometheus.go:75] 0.02
I0303 00:14:59.266337       1 prometheus.go:75] 0.33
...
I0303 00:16:58.143989       1 prometheus.go:75] 99.98
I0303 00:16:59.145063       1 prometheus.go:75] 99.99
I0303 00:17:00.145488       1 prometheus.go:75] 99.99
I0303 00:17:01.145856       1 prometheus.go:75] 99.99
I0303 00:17:02.147496       1 prometheus.go:75] 99.99
I0303 00:17:03.165908       1 prometheus.go:75] 100.00
I0303 00:17:04.153830       1 data-processor.go:255] New phase: Resize
I0303 00:17:04.167083       1 prometheus.go:75] 100.00
E0303 00:17:04.180605       1 prlimit.go:155] failed to kill the process; os: process already finished
W0303 00:17:04.181622       1 data-processor.go:338] Available space less than requested size, resizing image to available space 5368709120.
I0303 00:17:04.181722       1 data-processor.go:341] No need to resize image. Requested size: 5681173672, Image size: 5368709120.
I0303 00:17:04.181962       1 data-processor.go:261] Validating image
E0303 00:17:04.188340       1 prlimit.go:155] failed to kill the process; os: process already finished
I0303 00:17:04.188480       1 data-processor.go:255] New phase: Complete
I0303 00:17:04.188618       1 importer.go:223] Import Complete

I than Define a VM see definition below... and it attempts to clone the PV

Source pod log failing

root@rke2-s0:~# kubectl logs -f f3d3cae7-fa02-4e01-aca4-41512aa2710c-source-pod -n vm-images
VOLUME_MODE=filesystem
MOUNT_POINT=/var/run/cdi/clone/source
/var/run/cdi/clone/source /
UPLOAD_BYTES=5368729600
I0303 00:18:59.909560      10 clone-source.go:220] content-type is "filesystem-clone"
I0303 00:18:59.909936      10 clone-source.go:221] mount is "/var/run/cdi/clone/source"
I0303 00:18:59.909990      10 clone-source.go:222] upload-bytes is 5368729600
I0303 00:18:59.910054      10 clone-source.go:239] Starting cloner target
I0303 00:18:59.910232      10 clone-source.go:177] Executing [/usr/bin/tar cv -S disk.img]
I0303 00:19:00.594647      10 clone-source.go:251] Set header to filesystem-clone
I0303 00:19:00.912371      10 prometheus.go:75] 2.18
I0303 00:19:01.912428      10 prometheus.go:75] 7.60
I0303 00:19:02.912628      10 prometheus.go:75] 12.71
I0303 00:19:03.912777      10 prometheus.go:75] 18.34
I0303 00:19:04.912982      10 prometheus.go:75] 23.11
I0303 00:19:05.913059      10 prometheus.go:75] 26.60
I0303 00:19:06.913253      10 prometheus.go:75] 29.98
I0303 00:19:07.913393      10 prometheus.go:75] 34.27
I0303 00:19:08.913606      10 prometheus.go:75] 40.25
I0303 00:19:09.913749      10 prometheus.go:75] 44.74
I0303 00:19:10.913817      10 prometheus.go:75] 47.34
I0303 00:19:11.913923      10 prometheus.go:75] 50.38
I0303 00:19:12.914073      10 prometheus.go:75] 55.24
I0303 00:19:13.914228      10 prometheus.go:75] 61.63
I0303 00:19:14.914310      10 prometheus.go:75] 67.05
I0303 00:19:15.914998      10 prometheus.go:75] 72.78
I0303 00:19:16.915087      10 prometheus.go:75] 78.53
I0303 00:19:17.965801      10 prometheus.go:75] 83.82
I0303 00:19:18.965878      10 prometheus.go:75] 90.04
I0303 00:19:19.966007      10 prometheus.go:75] 95.90
I0303 00:19:20.574333      10 clone-source.go:127] Wrote 5368719360 bytes
F0303 00:19:20.577763      10 clone-source.go:260] Unexpected status code 500

cdi upload logs

root@rke2-s0:~# kubectl logs -f cdi-upload-tmp-pvc-1d761a41-fc76-47ea-9cad-f1c29b75bac2 -n vm-images
I0303 00:18:54.308174       1 uploadserver.go:74] Running server on 0.0.0.0:8443
I0303 00:19:00.611111       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:00.611315       1 util.go:207] begin untar to /data...
I0303 00:19:00.611347       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]
E0303 00:19:20.576544       1 util.go:222] exit status 2
E0303 00:19:20.576914       1 uploadserver.go:402] Saving stream failed: error unarchiving to /data: exit status 2
I0303 00:19:22.030210       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:22.030376       1 util.go:207] begin untar to /data...
I0303 00:19:22.030391       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]
E0303 00:19:42.279237       1 util.go:222] exit status 2
E0303 00:19:42.279336       1 uploadserver.go:402] Saving stream failed: error unarchiving to /data: exit status 2
I0303 00:19:57.195179       1 uploadserver.go:389] Content type header is "filesystem-clone"
I0303 00:19:57.195323       1 util.go:207] begin untar to /data...
I0303 00:19:57.195371       1 util.go:213] running untar cmd: [/usr/bin/tar --preserve-permissions --no-same-owner -xvC /data]

mounts where the Pod is attempting to write "disk.img"

root@rke2-a0:~# mount | grep pvc
/dev/zd16 on /var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount type ext4 (rw,relatime,stripe=4)
/dev/zd0 on /var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount type ext4 (rw,relatime,stripe=4)
root@rke2-a0:~# ls /var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount
lost+found
root@rke2-a0:~# ls /var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount
disk.img  lost+found
root@rke2-a0:~#

# The Source container
root@rke2-a0:~# crictl inspect  286881c923e1f | jq -r '.info.config.mounts'
[
  {
    "container_path": "/var/run/cdi/clone/source",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~csi/pvc-8da0c402-4527-4cc0-8fc5-c4a58d57725b/mount",
    "readonly": true
  },
  {
    "container_path": "/var/run/secrets/kubernetes.io/serviceaccount",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/volumes/kubernetes.io~projected/kube-api-access-dwvzc",
    "readonly": true
  },
  {
    "container_path": "/etc/hosts",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/etc-hosts"
  },
  {
    "container_path": "/dev/termination-log",
    "host_path": "/var/lib/kubelet/pods/8c644fbc-44a1-4959-a71b-b43373589331/containers/cdi-clone-source/07dee6b3"
  }
]

# Upload Container
root@rke2-a0:~# crictl inspect  992eb5929e7fa | jq -r '.info.config.mounts'
[
  {
    "container_path": "/data",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~csi/pvc-f3d3cae7-fa02-4e01-aca4-41512aa2710c/mount"
  },
  {
    "container_path": "/var/run/secrets/kubernetes.io/serviceaccount",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/volumes/kubernetes.io~projected/kube-api-access-4gtch",
    "readonly": true
  },
  {
    "container_path": "/etc/hosts",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/etc-hosts"
  },
  {
    "container_path": "/dev/termination-log",
    "host_path": "/var/lib/kubelet/pods/1726f4ba-3156-4d74-a110-19735298bb6f/containers/cdi-upload-server/472c045e"
  }
]

I have the following Defined: root@rke2-s0:~# kubectl version Client Version: v1.29.0+rke2r1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.0+rke2r1

VM Definition

root@rke2-s0:~# cat vm1_pvc.yml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: bruce
  name: bruce
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: bruce
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumedisk1
          - disk:
              bus: virtio
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            memory: 4Gi
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: bruce
        name: datavolumedisk1
      - cloudInitNoCloud:
          userData: |
            #cloud-config
            hostname: bruce
            ssh_pwauth: True
            disable_root: false
            ssh_authorized_keys:
            - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU6mGTN3ztmIwgd5sZt6AhXH7PidRc+yZ9OospfgBA7cRQaFoWavYcdw5F0vOEzt0USrlPZjxKPxOX2eoF98os3A3H4fp6+5LCkNnn+OZcbpkbf+53j0pNHvfH9X7FiyVez4F8v7uC7KWiBDKy1J3OB026bgMnpV3+PtKiC3zG0BQGcf/KN+QRZqk9qAAdEbSSUHtc+1wJEZDWVTjNREQGzZVn5F1pm4YkQz44WnD6wndsLJ9+e5vEscON3SlUujJPoOGKBu+uuxhjS5kPR5+hMJ3fjtfGCWYudWIE5ZLQI1LrD/7qfHDKtUyKD0eLSPtwuhFk7zJYuao6zKbZaEH/
        name: cloudinitdisk
  dataVolumeTemplates:
  - metadata:
      name: bruce
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 7Gi
      source:
        pvc:
          namespace: vm-images
          name: fedora

DataVolume Source

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "fedora"
  namespace: "vm-images"
spec:
  storage:
    resources:
      requests:
        storage: 5Gi
    storageClassName: "images"
  source:
    http:
      url: "https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"

Storage Profile

root@rke2-s0:~# kubectl get storageprofile images -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  creationTimestamp: "2024-03-02T05:53:07Z"
  generation: 3
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    cdi.kubevirt.io: ""
  name: images
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 02f7dc11-6ca6-4036-ab3a-08d5b71a5feb
  resourceVersion: "559411"
  uid: 9c584532-a0bd-42cf-bfe5-47f6c9c7c9ff
spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
status:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
  cloneStrategy: copy
  dataImportCronSourceFormat: pvc
  provisioner: zfs.csi.openebs.io
  storageClass: images

Storage Classes

root@rke2-s0:~# kubectl get storageclass -o yaml
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4SRP0/DMBTEvwq6OU2VtvlTSwyIkRGJ/Tl5bU0Sv8jPBEHV745MO2SBjvbd/XzWnUGTe+OgTjwMNEqgI+d9o7mT9VwgQ+98B4PXq/Q8kCoyjBypo0gwZ5D3Eik68ZqOYt+5jcoxD07ylmIcONFcwiD7U5dPz2F1nHsY9FtdKHORPbw43z0+dZ34uwhPI8PAjXTk9tb3fkInalOs/7C80i+NPOKSYSDLw7//OpGeYHCoinpPVWlLtjWXddOUm93BbnlfbXi7qwvu6qqiJkGXDTVdTBRo5Mjh96VJZLhZrkXWC2eQ2aW5OMDg+6B5qy6XiT3bNBouPwEAAP//DxMpHtQBAAA
      objectset.rio.cattle.io/id: ""
      objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
      objectset.rio.cattle.io/owner-name: imageclass
      objectset.rio.cattle.io/owner-namespace: kube-system
    creationTimestamp: "2024-03-02T05:51:51Z"
    labels:
      objectset.rio.cattle.io/hash: f6179a65b5eb7e5788524fb3e962e3471ed766a8
    name: images
    resourceVersion: "339"
    uid: f737dde3-589d-45f8-bc48-df295cfd570f
  parameters:
    poolname: system/images
  provisioner: zfs.csi.openebs.io
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"volumes"},"parameters":{"poolname":"system/volumes"},"provisioner":"zfs.csi.openebs.io"}
      storageclass.kubernetes.io/is-default-class: "true"
    creationTimestamp: "2024-03-02T23:45:52Z"
    name: volumes
    resourceVersion: "559539"
    uid: 169e19da-ec92-45c1-bdc1-32632332c3b0
  parameters:
    poolname: system/volumes
  provisioner: zfs.csi.openebs.io
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
kind: List
metadata:
  resourceVersion: ""

Kubernetes version

root@rke2-s0:~# kubectl version
Client Version: v1.29.0+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0+rke2r1

CDI Deployment


apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    operator.cdi.kubevirt.io/lastAppliedConfiguration: '{"kind":"Deployment","apiVersion":"apps/v1","metadata":{"name":"cdi-deployment","namespace":"cdi","creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"containerized-data-importer"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"containerized-data-importer","app.kubernetes.io/component":"storage","app.kubernetes.io/managed-by":"cdi-operator","cdi.kubevirt.io":"","prometheus.cdi.kubevirt.io":"true"}},"spec":{"volumes":[{"name":"cdi-api-signing-key","secret":{"secretName":"cdi-api-signing-key","items":[{"key":"id_rsa.pub","path":"id_rsa.pub"},{"key":"id_rsa","path":"id_rsa"}],"defaultMode":420}},{"name":"uploadserver-ca-cert","secret":{"secretName":"cdi-uploadserver-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-client-ca-cert","secret":{"secretName":"cdi-uploadserver-client-signer","items":[{"key":"tls.crt","path":"tls.crt"},{"key":"tls.key","path":"tls.key"}],"defaultMode":420}},{"name":"uploadserver-ca-bundle","configMap":{"name":"cdi-uploadserver-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}},{"name":"uploadserver-client-ca-bundle","configMap":{"name":"cdi-uploadserver-client-signer-bundle","items":[{"key":"ca-bundle.crt","path":"ca-bundle.crt"}],"defaultMode":420}}],"containers":[{"name":"cdi-controller","image":"quay.io/kubevirt/cdi-controller:v1.58.1","args":["-v=1"],"ports":[{"name":"metrics","containerPort":8080,"protocol":"TCP"}],"env":[{"name":"IMPORTER_IMAGE","value":"quay.io/kubevirt/cdi-importer:v1.58.1"},{"name":"CLONER_IMAGE","value":"quay.io/kubevirt/cdi-cloner:v1.58.1"},{"name":"UPLOADSERVER_IMAGE","value":"quay.io/kubevirt/cdi-uploadserver:v1.58.1"},{"name":"UPLOADPROXY_SERVICE","value":"cdi-uploadproxy"},{"name":"PULL_POLICY","value":"IfNotPresent"},{"name":"INSTALLER_PART_OF_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/part-of'']"}}},{"name":"INSTALLER_VERSION_LABEL","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.labels[''app.kubernetes.io/version'']"}}}],"resources":{"requests":{"cpu":"100m","memory":"150Mi"}},"volumeMounts":[{"name":"cdi-api-signing-key","mountPath":"/var/run/cdi/token/keys"},{"name":"uploadserver-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-signer"},{"name":"uploadserver-client-ca-cert","mountPath":"/var/run/certs/cdi-uploadserver-client-signer"},{"name":"uploadserver-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-signer-bundle"},{"name":"uploadserver-client-ca-bundle","mountPath":"/var/run/ca-bundle/cdi-uploadserver-client-signer-bundle"}],"readinessProbe":{"exec":{"command":["cat","/tmp/ready"]},"initialDelaySeconds":2,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["ALL"]},"runAsNonRoot":true,"allowPrivilegeEscalation":false,"seccompProfile":{"type":"RuntimeDefault"}}}],"nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"cdi-sa","securityContext":{"runAsNonRoot":true},"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"priorityClassName":"kubevirt-cluster-critical"}},"strategy":{}},"status":{}}'
  creationTimestamp: "2024-03-02T05:52:53Z"
  generation: 1
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-operator
    cdi.kubevirt.io: ""
    operator.cdi.kubevirt.io/createVersion: v1.58.1
    prometheus.cdi.kubevirt.io: "true"
  name: cdi-deployment
  namespace: cdi
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CDI
    name: cdi
    uid: 02f7dc11-6ca6-4036-ab3a-08d5b71a5feb
  resourceVersion: "1878"
  uid: 1b3aa410-8b11-4571-b02a-6631dc74e3f3
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: containerized-data-importer
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-operator
        cdi.kubevirt.io: ""
        operator.cdi.kubevirt.io/createVersion: v1.58.1
        prometheus.cdi.kubevirt.io: "true"
    spec:
      containers:
      - args:
        - -v=1
        env:
        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.58.1
        - name: CLONER_IMAGE
          value: quay.io/kubevirt/cdi-cloner:v1.58.1
        - name: UPLOADSERVER_IMAGE
          value: quay.io/kubevirt/cdi-uploadserver:v1.58.1
        - name: UPLOADPROXY_SERVICE
          value: cdi-uploadproxy
        - name: PULL_POLICY
          value: IfNotPresent
        - name: INSTALLER_PART_OF_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/part-of']
        - name: INSTALLER_VERSION_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/version']
        image: quay.io/kubevirt/cdi-controller:v1.58.1
        imagePullPolicy: IfNotPresent
        name: cdi-controller
        ports:
        - containerPort: 8080
          name: metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
          failureThreshold: 3
          initialDelaySeconds: 2
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        res      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-operator
        cdi.kubevirt.io: ""
        operator.cdi.kubevirt.io/createVersion: v1.58.1
        prometheus.cdi.kubevirt.io: "true"
    spec:
      containers:
      - args:
        - -v=1
        env:
        - name: IMPORTER_IMAGE
          value: quay.io/kubevirt/cdi-importer:v1.58.1
        - name: CLONER_IMAGE
          value: quay.io/kubevirt/cdi-cloner:v1.58.1
        - name: UPLOADSERVER_IMAGE
          value: quay.io/kubevirt/cdi-uploadserver:v1.58.1
        - name: UPLOADPROXY_SERVICE
          value: cdi-uploadproxy
        - name: PULL_POLICY
          value: IfNotPresent
        - name: INSTALLER_PART_OF_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/part-of']
        - name: INSTALLER_VERSION_LABEL
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/version']
        image: quay.io/kubevirt/cdi-controller:v1.58.1
        imagePullPolicy: IfNotPresent
        name: cdi-controller
        ports:
        - containerPort: 8080
          name: metrics
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/ready
          failureThreshold: 3
          initialDelaySeconds: 2
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:ources:
          secretName: cdi-uploadserver-signer
      - name: uploadserver-client-ca-cert
        secret:
          defaultMode: 420
          items:
          - key: tls.crt
            path: tls.crt
          - key: tls.key
            path: tls.key
          secretName: cdi-uploadserver-client-signer
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-signer-bundle
        name: uploadserver-ca-bundle
      - configMap:
          defaultMode: 420
          items:
          - key: ca-bundle.crt
            path: ca-bundle.crt
          name: cdi-uploadserver-client-signer-bundle
        name: uploadserver-client-ca-bundle
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2024-03-02T05:53:08Z"
    lastUpdateTime: "2024-03-02T05:53:08Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-03-02T05:52:53Z"
    lastUpdateTime: "2024-03-02T05:53:08Z"
    message: ReplicaSet "cdi-deployment-74b786dcc6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
akalenyu commented 5 months ago

Hey, thanks for reporting this!

I think if you increase the CDI log verbosity with https://github.com/kubevirt/containerized-data-importer/blob/release-v1.58/doc/debug.md#log-verbosity We should get the actual stdout/stderr of the untar command

May be a similar issue to https://github.com/kubevirt/containerized-data-importer/issues/2982

aglitke commented 3 months ago

@Smithx10 If you'd still like to pursue this issue could you please retry with more verbosity?

cccsss01 commented 2 months ago

I'm running into OOMKilled w/ 32Gi RAM

awels commented 2 months ago

I'm running into OOMKilled w/ 32Gi RAM

So the import/clone/upload pods have a request/limit set on them, and you get OOM killed if you go over that limit. See https://github.com/kubevirt/containerized-data-importer/blob/main/doc/quota.md on how to modify that limit. In most cases it should not be an issue, but we are seeing that slow storage has the ability to exhaust that memory because the OS is caching the writes in memory that counts against that limit.

cccsss01 commented 2 months ago

@awels Applying the example CDIConfig fails. error: error parsing cdiconfig.yaml: error converting YAML to JSON: yaml: line 9: did not find expected ',' or '}'

I've tried the example, and spec: podResourceRequirements: limits: cpu: "4" memory: "1Gi" requests: cpu: "1" memory: "250Mi" This fails with unknown field "spec.limits", unknown field "spec.requests"

PR submitted for what the CDI components that worked for me, however still testing the overall process

cccsss01 commented 2 months ago

I did get farther, however I received Error io: read/write on closed pipe piping to gzip on the source pod Error Post "https://cdi-upload-tmp-pvc-58b0c7a5-f12b-4b0f-9ef5-471bd95e5fde.default.svc/v1beta1/upload": write tcp 10.0.0.99:51356->10.111.44.190:443: write: connection reset by peer POSTing to https://cdi-upload-tmp-pvc-58b0c7a5-f12b-4b0f-9ef5-471bd95e5fde.default.svc/v1beta1/upload

cccsss01 commented 2 months ago

for vm1_pvc.yml I get failed to create virtual machine pod: pods "virt-launcher-vm1-r6rvd" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu for: compute; limits.memory for: compute

akalenyu commented 1 month ago

for vm1_pvc.yml I get failed to create virtual machine pod: pods "virt-launcher-vm1-r6rvd" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu for: compute; limits.memory for: compute

Is there quota/limitrange on the namespace? In that case, you would have to specify requests/limits on the VM resource

akalenyu commented 1 month ago

@cccsss01 hey, did you get a chance to try this since? specifically interested if there were some limits on the namespace

cccsss01 commented 1 month ago

There was no quota limitrange on the namespace, however, specifying the requests/limits worked.