"upload_ova.py" fails if there is an iso file inside the ova

[reubenur-rahman]

Hello @prziborowski

The upload_ova code is not working if there is an iso file inside the ova. I have a init script iso file packed with the ova.

I found that the vmdk got successfully uploaded in the datastore. But while uploading the iso I found the urlopen method is giving urlopen error.

However I can import the ova using vSphere client.

Given below is the output of the url:

DeviceUrl:
 (vim.HttpNfcLease.DeviceUrl) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   key = '/vm-10649/VirtualLsiLogicController0:0',
   importKey = '/My-VM/VirtualLsiLogicController0:0',
   url = 'https://10.10.10.10/nfc/52c1076f-5075-2d62-f47a-bc5f845c6c2c/disk-1.vmdk',
   sslThumbprint = '69:44:03:56:F3:F2:28:F5:3F:A3:72:1A:AD:C8:35:56:A0:E7:B7:7B',
   disk = true,
   targetId = 'disk-1.vmdk',
   datastoreKey = 'https://10.10.10.10/nfc/52c1076f-5075-2d62-f47a-bc5f845c6c2c/',
   fileSize = <unset>
}

URL:
 https://10.10.10.10/nfc/52c1076f-5075-2d62-f47a-bc5f845c6c2c/disk-1.vmdk

DeviceUrl:
 (vim.HttpNfcLease.DeviceUrl) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   key = '/vm-10649/VirtualIDEController1:0',
   importKey = '/My-VM/VirtualIDEController1:0',
   url = 'https://10.10.10.10/nfc/52c1076f-5075-2d62-f47a-bc5f845c6c2c/disk-0.iso',
   sslThumbprint = '69:44:03:56:F3:F2:28:F5:3F:A3:72:1A:AD:C8:35:56:A0:E7:B7:7B',
   disk = false,
   targetId = <unset>,
   datastoreKey = <unset>,
   fileSize = 43008L
}

URL :
https://10.10.10.10/nfc/52c1076f-5075-2d62-f47a-bc5f845c6c2c/disk-0.iso

[prziborowski]

Thank you for the report. Once I find an ova with an iso embedded, I'll try to fix this.

[prziborowski]

I'd be very curious about how vSphere client is doing this. When I try with ovftool, it seems to fail (can't find the iso...).

Anyway, from a lot of hacking around and looking at trivia logs, it seems like non-disks need to use PUT rather than POST. But by default PUT doesn't allow existing files (and the createSpec creates a blank file). Hence I'm curious how the UI client gets around this.

--- a/samples/deploy_ova.py
+++ b/samples/deploy_ova.py
@@ -293,7 +293,10 @@ class OvfHandler(object):
             sslContext = ssl._create_unverified_context()
         else:
             sslContext = None
-        req = Request(url, ovffile, headers)
+        if not deviceUrl.disk:
+            headers['Overwrite'] = 't'
+        method = 'PUT' if not deviceUrl.disk else 'POST'
+        req = Request(url, ovffile, headers, method=method)
         urlopen(req, context=sslContext)

     def start_timer(self):

I'll try to get this change in, but I noticed some other usability issues with older versions of python 2, as well as a progress update issue and I want to make sure those are resolved as well. Try out the above patch and see if that works for you.

[reubenur-rahman]

Cool it's working! I have also figured out PUT will work for non disk files when I went through the code upload_file_in_datastore. But I was struggling to find the overwrite header.

Thanks ! I am leaving the issue on you to close.

[reubenur-rahman]

@prziborowski One question ?

Trying to upload a remote ova. To read the ova I need to pass cookie in the header. So the only change I have done in WebHandle() class is added header with cookies to read the remote OVA.

WebHandle() class can successfully read the ova file but the upload is very very slow. It's taking 1 hour to complete 1%.

The OVA size is 600MB. So possibly the slowness is due to heavy size ! I tried to to use requests.get(stream=True) in the read() but no luck.

Would you please suggest any pointer that I can try ?

Thanks !

[prziborowski]

I haven't really looked into how to improve the performance of the remote ova process. 1 hour for 1% does seem pretty bad though (~6MB?). Usually I fall back to using the ovftool program if I don't need to script something in python around it (or even if performance is much better with ovftool then it might be worth it).

[revfran]

The overwrite header hint suggested here helped me overcome the problem of uploading an iso file, in my case using curl as in the deploy_ovf sample.

Thanks so much @prziborowski !

[bouchardh]

Sorry to post to sutch an old issue but I am experiencing the same problem. I am trying to deploy a Cisco ASAv with the example script and is failing on the iso file. One thing I noticed is that the lease.info variable has the following before even starting to upload files: (vim.HttpNfcLease.DeviceUrl) { dynamicType = , dynamicProperty = (vmodl.DynamicProperty) [], key = '/vm-723/VirtualIDEController0:1', importKey = '/ASAv/VirtualIDEController0:1', url = 'https://esx/nfc/52f615b5-6908-ed5c-f7b2-dd536dd9bbd4/disk-0.iso', sslThumbprint = 'BC:D6:80:A2:D8:ED:03:35:CA:98:BE:7B:E7:51:3E:EF:7F:DC:16:54', disk = false, targetId = , datastoreKey = , fileSize = 43008 } The actual file size is 350K and I know that the descriptor in CreateImportSpec has the proper info. Also, if I look at the datastore while this is hapening, all the files have already been created and the iso file is 42K in size. It looks like the connection to the server is forcibly closed after transfering 42K of data but I could be wrong.

[nvazquez]

@prziborowski I was having the same issue uploading an OVA with ISO through the Java SDK and your hit to add the Overwrite header worked perfectly! Thanks so much!

[vchennepalli]

@prziborowski, @reubenur-rahman, Any solution identified to improve the performance of the remote ova process?

[prziborowski]

@prziborowski, @reubenur-rahman, Any solution identified to improve the performance of the remote ova process?

Do you mean like https://github.com/vmware/pyvmomi-community-samples/pull/692 does? I would think reading a larger block size would help. The python script was more of an example of how it could be done and not really how it can be done in the fastest way.

If performance is key, then possibly python isn't the best language to use (ovftool I believe is much faster). Possibly it could also be multi threaded so the readers and writers work independently (unless that is how it works under the hood).

Also when I wrote the script I was naive (I still am) about OVA format. In addition to being packaged as a tar file, it is supposed to be ordered so be streamed, so it shouldn't have to use http Ranges when making requests as it should be able to read the file from beginning to end. Changing this would likely require a re-write though. And still http_block_size change in above PR most likely would benefit it.

[vchennepalli]

@prziborowski, thank you