upload a vmdk to vsan datastore leaves disk unusable

Klaas- commented 8 months ago

Describe the bug

Hi, uploading a vmdk with pyvmomi to a vsan datastore seem to leave the resulting file in an unusable state.

This pops up for me because I am using vsphere_copy in ansible (which is using the sdk in background).

For simplicity I reproduced the issue with the sample datastore upload python: https://github.com/vmware/pyvmomi-community-samples/blob/master/samples/upload_file_to_datastore.py

Version 8.0.2.0.1 Python 3.11 on a RHEL8.9

vmdk is a stream optimized v6 image, using the same file and uploading it via the webui through firefox results in a usable vmdk.

Uploading the vmdk via api results in an error when trying to use in vsphere: "Unsupported or invalid disk type 22 for 'scsi0:0'. Ensure that the disk has been imported."

vCenter: 7.0.3 Build: 22357613 (VxRail 7.0.482)

Reproduction steps

have a stream optimized vmdk + a vsphere + vsan
python3.11 upload_file_to_datastore.py -s vsphere_server -o 443 -u username@vsphere.local -p "password" --disable-ssl-verification --datastore-name VxRail-Virtual-SAN-Datastore-id --local-file-path ~/diskimages/file.vmdk --remote-file-path import/file.vmdk
create a VM that uses that VMDK as disk ...

Expected behavior

Works, no error

Additional context

Actual behavior: VM does not start; error "Unsupported or invalid disk type 22 for 'scsi0:0'. Ensure that the disk has been imported."

Klaas- commented 8 months ago

I figured it out, so not sure if this belongs here or somewhere else :)

While comparing what the sample/sdk does to what the WebUI does I noticed that the webui sets an additional argument that is not documented as far as I could find

So if you set the option like this:

        params = {"dsName": datastore.info.name,
                  "dcPath": datacenter.name,
                  "diskFormat": "StreamVmdk"
                 }

It works fine. Not sure if this is considered to be a problem in the sdk/example but it for sure is missing from the api documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/c369f363-3f34-4acb-8620-e861a6c24013/1c6e721f-2d3b-4740-bf94-3409efb5f08b/WebServicesSDKprogramming802.pdf

mariolenz commented 7 months ago

I didn't find any documentation about this diskFormat parameter. I didn't really find any documentation about uploading files at all.

It would be really great to see this officially documented somewhere ;-)

Klaas- commented 7 months ago

And a small follow up, there seems to be a second issue with python 3.10+ which is creating errors during large put requests ( happens often like 9/10 times - https://github.com/python/cpython/issues/110467 ), this also seems to be happening on 3.9 but not as often :) so now I just use python 3.9 and I do automatic retries if upload fails :)

Klaas- commented 7 months ago

and to document my workaround for the python file upload issue: I now just execute curl, it seems to have a 100% success rate :)

curl -u "user:pass" "https://vsphere.tld/folder/DIRNAME/FILENAME.vmdk?dsName=VxRail-Virtual-SAN-Datastore-UUID&dcPath=DATACENTERNAME&diskFormat=StreamVmdk" -H 'Content-type: application/octet-stream' -T /path/to/local.vmdk

vmware / pyvmomi