OVA image does not accept cloud-init userdata

[johanneskastl]

Problem description

With @schaefi 's help I was able to build a OVA image of openSUSE Leap 15.5 on OBS. However I cannot deploy a virtual machine using this OVA file in vSphere via Terraform/OpenTOFU, as the cloud-config / cloud-init userdata and metadata elements are not accepted. So the VM is not created successfully

Expected behaviour

The OVA image should allow customizing it with cloud-config and should "say" that in its ovf configuration file. I checked the file inside the OVA image, but only found hardware configuration and information on the disk.

I'll compare this with a Fedora CoreOS image that accepts ignition configuration.

Steps to reproduce the behaviour

Using vSphere, I could upload the OVA file into vSphere's content library. From there I tried to deploy a VM following the Terraform vSphere provider documentation:

https://registry.terraform.io/providers/hashicorp/vsphere/latest/docs/resources/virtual_machine#creating-a-virtual-machine-from-a-template

Normally the cloud-init metadata and userdata go into guestinfo.metadata and guestinfo.userdata keys respectively.

Something like this:

resource "vsphere_virtual_machine" "vm" {
  # ... other configuration ...
  clone {
    template_uuid = data.vsphere_virtual_machine.template_from_ovf.id
  }
  vapp {
    properties = {
     "guestinfo.metadata"          = base64encode(file("metadata.yaml"))
     "guestinfo.metadata.encoding" = "base64"
     "guestinfo.userdata"          = base64encode(file("userdata.yaml"))
     "guestinfo.userdata.encoding" = "base64"
    }
  }
}

OS and Software information

The image was built on build.opensuse.org with whatever versions are used there. The project is here: https://build.opensuse.org/project/show/home:ojkastl_buildservice:branches:Virtualization:Appliances:Images:openSUSE-Leap-15.5

[johanneskastl]

The Fedora OVA file contains something like this (for ignition obviously, but you get the idea):

[...]
    </VirtualHardwareSection>                                                                                                                                                                                                                                                            
    <ProductSection>                                                                                                                                                                                                                                                                     
      <Info>Information about the installed software</Info>                                                                                                                                                                                                                              
      <Product>fedora-coreos Fedora CoreOS stable</Product>                                                                                                                                                                                                                              
      <Vendor>fedora-coreos</Vendor>                                                                                                                                                                                                                                                     
      <Version>39.20240112.3.0</Version>                                                                                                                                                                                                                                                 
      <Property ovf:userConfigurable="true" ovf:type="string"                                                                                                                                                                                                                            
                ovf:key="guestinfo.ignition.config.data" ovf:value="">                                                                                                                                                                                                                   
        <Label>Ignition config data</Label>                                                                                                                                                                                                                                              
        <Description>Inline Ignition config data</Description>                                                                                                                                                                                                                           
      </Property>                                                                                                                                                                                                                                                                        
      <Property ovf:userConfigurable="true" ovf:type="string"                                                                                                                                                                                                                            
                ovf:key="guestinfo.ignition.config.data.encoding" ovf:value="">                                                                                                                                                                                                          
        <Label>Ignition config data encoding</Label>                                                                                                                                                                                                                                     
        <Description>Encoding for Ignition config data</Description>                                                                                                                                                                                                                     
      </Property>                                                                                                                                                                                                                                                                        
    </ProductSection>                                                                                                                                                                                                                                                                    
  </VirtualSystem>                                                                                                                                                                                                                                                                       
</Envelope> 

[schaefi]

I think the settings file created by kiwi is either incomplete or inappropriate regardring the VSphere you are deploying to. So to make this a bit more clear. When kiwi creates the ovf it calls:

EXEC: [/usr/bin/ovftool --shaAlgorithm=SHA1 /usr/src/packages/KIWI-vmx/openSUSE-Leap-15.5-Minimal.x86_64-15.5.0.vmx /usr/src/packages/KIWI-vmx/openSUSE-Leap-15.5-Minimal.x86_64-15.5.0.ova]

That .vmx file is the VMware settings file which is consumed by ovftool and turns things into an ovf. My expectation was when ovftool completes successfully the result is accepted in the VMware ecosystem... wrong assumption

So additional information needs to be provided and at that point we had no resources to follow VMware. Regarding the possible settings I added some documentation here:

• https://osinside.github.io/kiwi/building_images/build_simple_disk.html#customizing-the-virtual-machine

Please be aware this is far from being complete, I assume it's only a small part of what can be described and this is also where the open-vmdk project kicks in.

kiwi does not support at all the mentioned <ProductSection> which might be a reason why it did not got accepted.

As we cannot follow what VMware does we added some raw content option. So you can add the following to your image description

<type ...>
    <machine>
        <vmconfig-entry>text</vmconfig-entry>
    </machine>
 </type>

The vmconfig-entry element is used to add entries directly into the virtual machine's configuration file. This is currently only supported for the `vmdk` format where the provided strings are directly pasted into the .vmx file.

So I hope with additional vmconfig-entries you can add the information that is missing such that your VSphere accepts it

Please note all of VMware is pretty proprietary stuff and we always had a hard time to establish an open communication channel with VMware and its products. That made it not easy to support their product supply chain in a user friendly way and offers quite some roadblocks.

We help as much as we can but there are limitations regarding VMware support. Hope this makes sense to you too

[schaefi]

Here is some information on the settings of the .vmx file:

• https://sanbarrow.com/vmx.html

Also see the examples: https://sanbarrow.com/vmx/vmx-examples.html As I said some sections kiwi supports natively others you need to add by custom vmconfig-entry's

All this is for ESX and I have to admit that I never tested anything on VSphere

[schaefi]

any news ?

[johanneskastl]

Sorry Marcus, I was not notified of your first answers for $REASONS. I'll try to find some time to test this the week after next and report back.

[johanneskastl]

OK, I finally had some time to dig into this again. I found no hint of either cloud-config, cloud-init, userdata or ignition in the links you gave.

I'll see if I can use the vmconfig-entry element to inject something.

[johanneskastl]

OK, I could not simply add the lines from the CoreOS ovf file to the one inside the openSUSE OVA.

I tried to use the vmdk from the OBS-built OVA with an adapted version of the FedoraCoreOS OVF-File. But whatever I do, vSphere rejects the resulting OVA. As the ovf files look very different I think I'll cease my experiments and wait until #2292 is ready to be used.

[schaefi]

OK, I could not simply add the lines from the CoreOS ovf file to the one inside the openSUSE OVA.

I tried to use the vmdk from the OBS-built OVA with an adapted version of the FedoraCoreOS OVF-File. But whatever I do, vSphere rejects the resulting OVA. As the ovf files look very different I think I'll cease my experiments and wait until #2292 is ready to be used.

ok, thanks for the feedback. That actually helps to prioritize the move to open-vmdk. At the moment I won't have time to jump on it though

[schaefi]

closing with reference to #2292