vcd_vapp_vm guestinfo/OVF metadata is cleared on CPU/Memory change

[wandera-petr-cerny]

Terraform Version

Terraform v0.13.2 provider registry.terraform.io/-/vcd v2.9.0 provider registry.terraform.io/terraform-providers/vcd v2.9.0

Affected Resource(s)

• vcd_vapp_vm

Terraform Configuration Files

resource "vcd_vapp_vm" "instance" {
  name          = "test"
  computer_name = "test"
  template_name = "Ubuntu18"
  vapp_name     = "test"
  memory        = 1024
  cpus          = 1
  cpu_cores     = 1

  customization {
    enabled = true
    force   = false
  }

  network {
    type               = "org"
    name               = "test"
    adapter_type       = "VMXNET3"
    ip_allocation_mode = "POOL"
    is_primary         = false
  }
}

Debug Output

https://gist.github.com/wandera-petr-cerny/99a93d8ef2cebbd9ace05bfbd0dc7d9c

Expected Behavior

When adding memory (ie. from 1024MB to 1028MB) or CPUs, the instance may reboot, but after reboot, customization should not be broken and the instance should have hostname and IPs configured.

Actual Behavior

After adding 1 CPU and running terraform apply, the instance is rebooted, starts with the additional core, but customization is broken. After logging into the instance via remote console and running vmware-rpctool 'info-get guestinfo.ovfEnv', the PropertySection is missing completely as if it was removed by the vcd provider. After powering off the VM and forcing re-customization the customization works again and running vmware-rpctool 'info-get guestinfo.ovfEnv' returns XML document that contains PropertySection with hostname and IP details.

Steps to Reproduce

1. Run terraform apply for the instance built from Ubuntu 18 template with open-vm-tools 11.0.5 and cloud-init 20.2.
2. Log into the machine and check with ip a that the interface assigned to the instance has IP address configured. Run vmware-rpctool 'info-get guestinfo.ovfEnv' and verify that the result contains PropertySection element.
3. Change vcd_vapp_vm resource to either add CPU, or 4MB of RAM. Run terraform apply again.
4. Log into the instance over remote console, check with ip a that the IP configuration is missing. Check with vmware-rpctool 'info-get guestinfo.ovfEnv' that there is no PropertySection from which open-vm-tools read network settings and pass them to Cloudinit that usually configures Netplan.
5. Power off the instance, power on with forced re-customization
6. Log into the machine - it is customized correctly again and IP configuration is fine

Important Factoids

vCloud Director version: 9.7.0.16081827

[Didainius]

Thanks @wandera-petr-cerny for the issue. One thing to check. I see you don't have any guest_properties set in your configuration so checking - did the VM come with some configuration from template? Or did you deploy it via UI and filling in ovfEnv?

Ideally, could you share the debug output as per the issue documentation? https://github.com/vmware/terraform-provider-vcd/blob/master/.github/ISSUE_TEMPLATE.md#debug-output

Thanks!

[wandera-petr-cerny]

Hi @Didainius, i have tried not setting guest_properties at all, setting the attribute to {} and setting it to a dummy value ({testing_fix = "keep this"}), but it made no difference. How can I check what configuration is already in the template? Do you mean the descriptor.ovf from the OVA file? I have added the link to the debug output into issue description. I built the instance with 1024MB RAM, enabled logging into the file and ran terraform apply to increase instance memory to 1028MB. The log contains that run only. If you want to get more details, please let me know. Thanks for looking into this!

[jpbuecken]

Hello, we faced the same issue, but found it is a VCD bug, not a provider issue.

We tested with VCD 10 and VCD 10.1 with the HTML5 GUI. Unfortunately I could not open an issue at VMware yet, something went wrong with my myvmware invitation. But I can share my prepared steps to reproduce:

The new cpu and memory hot add feature in master for the next provider version may mitigate the issue, since it avoids power cycle:

Bug: OVF PropertySection is missing after PowerCycle

Steps to Reproduce:

Create a linux VM (we used CentOS 7 template) with

• open-vm-tools of the OS pre-installed
• PowerOff (disable checkbox poweron in UI)
• no need to connect a real network, I used none in the interface settings

Click Create

Go to the guest customization settings of the VM and make sure the following is set:

• Enable Guestcustomization ON
• Allow local administrator password
• Add a password, we will use it later

Click PowerON and Force Guestrecustomization

Wait some minutes until guest customization is complete (or check API Call for guest customization that should show GC_COMPLETE)

Open the VM console and login to the linux VM as root Check OVF Environment via:

/usr/bin/vmware-rpctool "info-get guestinfo.ovfEnv"'

Check that a line

   <Property oe:key="vCloud_computerName" oe:value="..."/>

does exists.

PowerOff the VM PowerOn the VM again (do not force guest customization)

Login to the linux vm as root Check OVF Environment again:

/usr/bin/vmware-rpctool "info-get guestinfo.ovfEnv"'

Now the line

   <Property oe:key="vCloud_computerName" oe:value="..."/>

and all other PropertySection is missing! -> BUG

[wandera-petr-cerny]

@jpbuecken thanks for the nice minimal working example. Once you are able to open a bug (and if it is on a publicly available tracker), could you please share a link so that I can watch it? Thanks!

[jpbuecken]

@jpbuecken thanks for the nice minimal working example. Once you are able to open a bug (and if it is on a publicly available tracker), could you please share a link so that I can watch it? Thanks!

Hi, I belong to a company that is a customer of VMware. So I can only use my organisation account / contract, but this is not public. Anyway it may makes sense if you open a case at VMware as well, such that VMware can prioritize the issue. I think it is more convincing if more than one VMware customer has the same issue / use case.

[jpbuecken]

For you information, I just opened a case add VMware. SR 20160961510

[vbauzys]

Latest comment in SR that it's fixed in 10.2 version.

[jpbuecken]

Unfortunately, after further tests of the vmware support, it is not fixed in 10.2. After last statement the developer are aware of this behavior but it is not decided yet if they are willing to fix it. It looks like we are the first customer who opened a case for this. So I really recommend to open cases yourself such that they see it is a used feature. I like the idea to use cloud-init / ovf / guest properties to configure cloud VMs. With the current behavior, there is no workaround for the missing metadata service similar to azure/aws/gcp/vio.

Anyway, it is not a terraform provider issue.