F #5: Implement multi-disk for hybrid transfer mode

[onenhansen]

This allows multiple disks for "hybrid" mode, where it uses rbvmomi2's Datastore.download method to cURL download the disk images locally, then creates a basic libvirt XML file to combine the disks, and converts the virtual machine locally. This should be a faster method for transferring the disks, and then the conversion should be quick as well since it's not making an nbd overlay over the network

This should work good for relatively simple VMs. I'm sure the XML generation could be expanded with certain options to improve it's chances of success.

@bishopbm2 - Can you run a couple tests on this? You should just need enabled :hybrid: true and if you're using the ESXi SSH transfer, comment out the ESXi auth section.

[bishopbm1]

Yep! We are messing with our network configs today so need to run a few tests anyway.

[tinova]

LGTM, let's hold the merge until @bishopbm1 (thanks!!) has time to report back

[gsperry2011]

I am testing this out today

[gsperry2011]

We have ran a few tests of this on the same VMware environment and same OpenNebula environment for this. We are actually seeing slightly slower speeds with --hybrid. Here is our latest test. Both of these are converting the same VM from VMware.

Using hybrid:

real    23m7.625s
user    1m20.908s
sys 2m9.239s

If it helps here is the xfer rate of the cURL:

Downloading disks from vCenter storage to local disk
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50.0G  100 50.0G    0     0  52.6M      0  0:16:12  0:16:12 --:--:-- 47.6M

I am not sure why we are only getting ~50M throughput here. I verified both our vcenter and OpenNebula frontend are connected over a 10G link so I would expect it to go faster. I suspect that the network is not the bottleneck here.

Not using hybrid, so we're setting ESXI_OPTS to do the transfer:

real    19m47.696s
user    0m8.390s
sys 0m15.456s

[onenhansen]

@gsperry2011 Thanks for testing this Greg.

That's interesting the curl speed is slower, I'm seeing speeds up to 250Mbps when doing testing, perhaps there are some other settings which can improve this. For my tests, the default and ESXi SSH transfer both seem to be slower. Definitely depends on the environment I guess, I'll maybe expand on this in the wiki!

It seems like you didn't see any errors though, so I'm going to merge this. It's at least another option just in case it is faster for some people.