Longhorn and contentType 'archive' don't work together

[ianb-mp]

What happened:

When I use contentType: archive together with Longhorn storage class for the PVC, my VM consistently fails to launch with error:

Warning  SyncFailed        115s (x16 over 4m39s)  virt-handler               preparing host-disks failed: unable to create /var/run/kubevirt-private/vmi-disks/datavolumedisk1-xxxx/disk.img, not enough space, │
│  demanded size 67108864 B is bigger than available space 57167872 B, also after taking 10 % toleration into account     

What you expected to happen:

Using archive type and Longhorn together should be possible.

How to reproduce it (as minimally and precisely as possible): Steps to reproduce the behavior.

Additional context:

• It doesn't matter what size I set the PVC to, I get the same error with the available space just a bit lower than what I requested.
• archive works fine if I use a storage class other than longhorn e.g. local-path
• Longhorn storage works fine for any data source other than archive: e.g. I use another PVC as the source; or I use HTTP source but specify a qcow2 image rather than tar file, comment contentType: archive, and adjust storage size etc.

Environment:

• CDI version (use kubectl get deployments cdi-deployment -o yaml): v1.59.0

• Kubernetes version (use kubectl version): v1.27.10+rke2r1

• DV specification:

  dataVolumeTemplates:
  - metadata:
    name: vm-archive-test
  spec:
    source:
        http:
           url: "http://10.2.3.2/rocky/archive_test/archive2.tar"
    contentType: archive
    pvc:
      #storageClassName: local-path
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 64Mi

  (Longhorn is set as the default storage class)

• Cloud provider or hardware configuration: bare metal, x86_64

• OS (e.g. from /etc/os-release): Rocky Linux 9.3

• Kernel (e.g. uname -a): 5.14.0-362.18.1.el9_3.x86_64

• Install tools: N/A

• Others: N/A

[awels]

So archive is supposed to be for non kubevirt disk image files. Not sure how it 'works' with other storage classes. Can you elaborate on your use case a bit. I would not expect it to work at all

[ianb-mp]

My use case is looking for a way to provide 'bootstrap' files to a VM that doesn't involve creating an ISO or mounting NFS etc. It wasn't clear to me what/how archive worked, so was experimenting with it to see if it might provide a solution to that. I was surprised to see it didn't work on Longhorn when, seemingly, every other scenario did work.

So archive is supposed to be for non kubevirt disk image files

IMHO the doco isn't clear on this. The example manifest here https://github.com/kubevirt/containerized-data-importer/blob/main/manifests/example/import-archive-datavolume.yaml shows a tar file containing arbitrary data (Geolite data, not a disk image!?).

[awels]

Right archive is for arbitrary files to be restored onto a filesystem PVC. The example you gave it looks like you are trying to boot of the contents of the PVC which won't work. If you want to get some arbitrary files into a VM from a PVC, I would suggest looking into the virtio-fs support provided by KubeVirt. It will simply make the volume available in the VM, and you can mount it whereever you want. If the data you want to provide is small like configuration, you can also directly mount config maps and secrets in a VM, and make them visible in the VM as well.

[ianb-mp]

Thanks for the suggestions, I'll look into those.

Regarding the Longhorn issue, I did some more troubleshooting and see that when I use local-path storage there is a directory on the host with the unpacked content of the archive and a new file disk.img (this wasn't contained in the archive tar):

$ ls -l /data/local-path-provisioner/pvc-f01a72b9-94a3-46d3-a2a9-b80c88cc7cad_default_vm-bootstrap-iso-xxxxx
total 4
drwxr-xr-x. 2 qemu root       28 Apr  5 15:33 blah
-rw-r--r--. 1 qemu qemu 67108864 Apr 10 11:38 disk.img
drwxr-xr-x. 2 qemu root       10 Apr  5 15:33 foo

If I add a 500KiB disk.img to the archive tar and try again, I see this instead:

$ ls -l /data/local-path-provisioner/pvc-32c28559-8669-443b-8294-d73732a17255_default_vm-bootstrap-iso-xxxxx
total 500
drwxr-xr-x. 2 qemu root     20 Apr  5 15:33 blah
-rw-r--r--. 1 qemu root 512000 Apr 10 11:11 disk.img
drwxr-xr-x. 2 qemu root      6 Apr  5 15:33 foo

And if I switch back to Longhorn storage and try again with this new archive file, it works fine (I don't get the disk space error).

Based on that, this is what I conclude:

• if disk.img is not found, an empty one will be created matching the size of the PVC
• this is a problem for Longhorn as the storage limit is more strict than local-path (I guess) and there isn't sufficient space to create a full size disk.img once the tar file has been unpacked.

If the above is correct, then I think CDI should be modified in these ways:

• update doco to make it clear that archive tar can contain arbitrary files, however it must contain disk.img in the root of the archive
• trying to use an archive that is missing disk.img should cause the archive unpacking step to fail

[awels]

So if it is creating a blank disk.img that would be a bug, since it should just unpack the contents of the archive into the PVC. It should not be creating a disk.img file.

[ianb-mp]

So if it is creating a blank disk.img that would be a bug, since it should just unpack the contents of the archive into the PVC. It should not be creating a disk.img file.

My understanding of how disks work with qemu is that a) it needs a disk image b) it will need a filesystem of some kind if the guest OS is going to be able to read data from it. So my expectation of how archive might work was that a disk image would be created inside the PVC with a filesystem (ext4 perhaps?) and then the archive unpacked into the disk image. But that's definitely not what I'm seeing - the disk image is zero'd out.

[awels]

The archive content type was designed to simply get the contents of an archive into the cluster and extracted onto a filesystem PVC. And then to be used with either a regular pod, or use the virtio-fs feature of kubevirt to make it visible inside a VM.

[ianb-mp]

or use the virtio-fs feature of kubevirt to make it visible inside a VM.

This is what I was missing. I've been trying to attach using a Kubevirt disk device, not filesystem (I wasn't aware of virtiofs prior to this). It all makes sense now - I'll close this issue

Do you think this update to README would be appropriate?

The archive content type indicates that the data is a tar archive. Compression is not yet supported for archives. CDI will extract the contents of the archive into the volume; which can then be used with either a regular pod, or a VM using Kubevirt's virtiofs feature.

[awels]

Yes, feel free to open a PR that does that. Anything that makes things more clear to users is helpful.