packer 1.3.x "bios.hddorder" in vmx causing trouble with ovftool

[as-dg]

Hi,

I've been using packer successfully up until version 1.2.x. Since packer version 1.3.0 there appears to be a change causing issues in my environment.

Host platforms on which I run packer: macOS (10.12.6): VMware Fusion Professional Version 8.5.10 (7527438) Ubuntu 16.04.2 LTS (4.4.0-62-generic): VMware Workstation Pro 14.1.1.7528167

So I build my virtual machines using the vmware-iso packer builder on those systems. Once completed, I deploy them to a vCenter/vSphere/ESXi environment via ovftool: ovftool --name="my-machine" --datastore="myDataStore" myVirtualMachine.vmx "vi://user@domain.local@vcenter-01.domain.local/Datacenter/host/Cluster"

This would then usually proceed and display something like:

Opening VMX source: /path/myVirtualMachine.vmx
Opening VI target: vi://user%domain.local@vcenter-01.domain.local:443/Datacenter/host/Cluster
Deploying to VI: vi://user%40domain.local@vcenter-01.domain.local:443/Datacenter/host/Cluster

Disk progress: 1%
Disk progress: 2%
Disk progress: 3%
[... shortened ...]
Disk progress: 98%
Disk progress: 99%
Transfer Completed                    
Completed successfully

The machine is then present in the remote environment and I can boot it up successfully.

However, with a packer-1.3.x built virtual machine, the upload process appears to be much faster, it completes right away. When attempting to boot up the virtual machine, it just attempts a network boot and goes on with displaying: "Operating System not found".

I compared the VMX file of a packer-1.2.x built VM with one created using packer-1.3.x and noticed that the latter one contains the following parameter: bios.hddorder = "scsi0:0"

After removing this parameter, everything was working as expected again for the packer-1.3.x built VM.

I can work around this by specifying the following in my packer template, leading to the parameter being removed after build:

"vmx_data_post": {
   "bios.hddorder": ""
}

Checking some of the packer code/history, I can see that the parameter has been added to the vmx template not too long ago in context of #6197 and https://github.com/hashicorp/packer/pull/6204

Any idea why this is causing ovftool to behave strangely and not upload the disk successfully? The resulting disk in my datastore is about 36MB while the original size was 1.5GB. No error thrown by ovftool though.

Thanks!

[zpratt]

I have confirmed this works for me as well and I appreciate the fix. It would be nice to not have to wait ~6 weeks to cut a release.

[SwampDragons]

@zpratt Sorry, but we don't expedite releases for bugfixes unless it's a critical bug or security issue, which I don't think this qualifies as, especially since you have a patched build you can use in the meantime. Also, HashiCorp shuts down for the holidays starting on Friday and doesn't reopen until early January, so cutting a release this week is a Bad Idea from a support perspective. Thanks for your patience though! I know it's frustrating to have to wait for something that ideally never would have broken in the first place.

[aaronk1]

Appears others are having same issue. I faced the same. I agree with what others have said, the upload definitely works with ESXi 6.7 and does not work with ESXi 6.5 for me. ovftool and PowerCLI Import-vApp produce same behavior, one of my two disks doesn't upload at all--totally ignored by both tools.

https://communities.vmware.com/message/2827714#2827714

[joedoss]

I believe I have a different build environment from others that have posted, but I experienced the same issue: packer 1.3.3 running on Windows 2016, using a remote ESXi 6.5 host for builds. "type": "vmware-iso", "disk_type_id": "zeroedthick", "disk_adapter_type": "lsisas1068", "remote_type": "esx5", "format": "ova", ovftool 4.3.0 Update 1 (same symptom was seen with ovftool 4.2.0)

The resulting OVA appeared to contain the correct vmdk, but the upload from that OVA would create an empty vmdk in the VM -- tried direct to ESXi 6.5 and through vcenter 6.5 to a different 6.5 host.

The windows patch build posted above from #7108 also resolves the issue for me.

[arizvisa]

Okay.. this issue has been closed some time ago with the solution and testing already done. So, please stop adding to the discussion unless you're trying to re-open the issue...

For the record, ESXi 6.5 is the _only_ version that has issues when receiving a VM that has the bios.hddorder option set. If you're using ESXi 6.7, VMware has already fixed this problem. It is up to them if they want to backport the patch for your ESXi 6.5 instance. If you have a support relationship with them, ask them about it.

So for people stuck with a bugged instance of ESXi 6.5:

1. If you need to deploy _anything_ to ESXi 6.5, you'll need to modify the .vmx and remove the bios.hddorder assignment. This option is not commonly used but was needed in certain instances where VMware will try and boot up off of the cdrom device (due to the bus type) instead of the hard disk device.

2. If (and only if) you have this aforementioned cdrom booting issue on ESXi 6.5, upload your VM without bios.hddorder set, and then modify the bootoptions attribute (VirtualMachineConfigInfo) via the web interface or by modifying the VM's config via PowerCLI.

3. If you don't know how to modify your .vmx (prior to uploading) and absolutely need to do this from within packer but can't wait for the new release, you can likely use vmx_data to set bios.hddorder to an empty string.

4. If none of these solutions satisfy you and you can't wait for a new release. @SwampDragons included a binary for each of the 3 main platforms in the comments (here). If not good enough, feel free to build your own release of packer that includes PR #7108.

But just so it's clear, the correct fix is to upgrade your instance of ESXi since your ESXi 6.5 instance doesn't support bios.hddorder and every other version of ESXi seems to.

It also appears that ovftool let's you specify properties via --prop which will let you affect the boot options. I personally don't use ovftool for anything as powershell-core and PowerCLI are more powerful for deployment. But I'd recommend looking into this if you're an ovftool fan...

(edited to add mention of the ovftool properties)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.