virtctl image-upload failed: A larger PVC is required

[cszixin]

use virtctl to upload a iso file, but always fails

 virtctl image-upload --image-path=/opt/app/gonghang2/cyber-range/server/data/upload/os_images/3a72c25408ac8fc7dd7e2395dd95785e           --pvc-name=61652d59223011cb5d611 --pvc-size=1194Mi --uploadproxy-url=https://10.111.91.61 --insecure --access-mode=ReadWriteMany -n gonghang2-vir --block-volume
PersistentVolumeClaim gonghang2-vir/61652d59223011cb5d611 created
Waiting for PVC 61652d59223011cb5d611 upload pod to be ready...

Pod now ready
Uploading data to https://10.111.91.61

 918.00 MiB / 918.00 MiB [=========================================================================================================================================================] 100.00% 3s

unexpected return value 400, Saving stream failed: Virtual image size 2147483648 is larger than available size 1251999744 (PVC size 2147483648, reserved overhead 0.000000%). A larger PVC is required.

use qemu-img cmd , it shows

{
    "virtual-size": 962592768,
    "filename": "/opt/app/gonghang2/cyber-range/server/data/upload/os_images/3a72c25408ac8fc7dd7e2395dd95785e",
    "format": "raw",
    "actual-size": 962592768
}

it so strange, why cdi -upload-proxy calculate Virtual image size 2147483648

there are versions info 

virtctl  version
Client Version: version.Info{GitVersion:"v0.39.0", GitCommit:"6493f7130788fc2242f0ebbda2ace06dc12a4ad0", GitTreeState:"clean", BuildDate:"2021-03-10T15:29:59Z", GoVersion:"go1.13.14", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v0.37.1-dirty", GitCommit:"f270b5befe16c20a11c34f22efecf2407f9591bc", GitTreeState:"dirty", BuildDate:"2021-01-21T20:13:15Z", GoVersion:"go1.13.14", Compiler:"gc", Platform:"linux/amd64"}

k8s: v1.19.2

cdi: v1.33.0

how to fix it?

[mazzystr]

@awels ping

[mazzystr]

@brybacki ping

[awels]

I CDI calls qemu-img info on the uploaded image to determine the virtual size of the image that is uploaded. I am surprised that your command gives the virtual size is the same as the actual size. This only happens if you have pre-allocated the entire disk. Can you tell us how big you are making the PVC? I suspect it is 2Gi, and I also suspect that the volume you are using is some kind of shared storage (local volume or hostpath, maybe nfs), and there simply is not that much space available.

[cszixin]

@awels thanks ! this is a centos image iso file with raw format, pvc size are --pvc-size=1194Mi, pvc volume are using ceph shared storage it is strange，No matter how large PVC size is specified, errors always exist, example pvc-size=2Gi, cdi shows virtual size is 3Gi, when pvc-size=3Gi, cdi shows virtual size is 4Gi i want to upload this iso file to k8s cluster and then it can be mount by kubevirt vms, but upload action always failed

[cszixin]

What makes me wonder is why for the same file, just because different PVC sizes are specified, CDI shows that the size of virtual size will change

[awels]

So at some point during the process CDI will attempt to resize the disk image to fit into the PVC. Which is where that size comes from. You say you are using ceph, are you using ceph rbd with a filesystem volume mode? If so you need to specify some fsOverhead so that CDI can take that into account when resizing. Basically what happens if you specify a 2Gi PVC, ceph-rbd will create a 2Gi block device, then put a file system on it. And the actual available space on the filesystem with be < 2Gi by a certain percentage. If you are using ceph-rbd, I suggest using block volume mode which doesn't have this problem and in general will perform better, and will allow you to create RWX volumes

[cszixin]

@awels volume mode is block , should i change it to filesystem? here is my cmd

virtctl image-upload --image-path=/opt/app/data/upload/os_images/3a72c25408ac8fc7dd7e2395dd95785e --pvc-name=61652d59c9793ae011cb5d61 --pvc-size=1194Mi --uploadproxy-url=https://10.111.91.61 --insecure --access-mode=ReadWriteMany -n gonghang2-vir --block-volume

[awels]

So an image upload will create a 2 PVCs, 1 scratch space which is file system RWO PVC, and the target PVC, which will be the block volume. The upload will then store the data in the scratch space, inspect it, then convert it and write it to the target. I suspect your scratch space gets created on the same storage class as the target (it will pick the default storage class unless overriden). Since that is rook-ceph, it will have the fs overhead issue I talked about above. So I think if you configure some reasonable overhead, it should all work.

[kubevirt-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to [this](https://github.com/kubevirt/kubevirt/issues/6586#issuecomment-1066923811): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.