Parallelly cloning multiple instances from a single template results in network customization errors

[fiskhest]

Using ansible, I have a task where I'm launching three guest instances from a stored template. The task is run in parallell, so the API is processing these three requests at roughly the same time(give or take a couple of seconds).

Whenever I do this on a completely clean re-run (no pre-reqs in place), the first VM is provisioned correctly and ansible reports that the inventory_host for that task has changed. However, the other two guest instances hangs indefinitely waiting for network customization to finish until the ansible timeout (3600s) is hit. These two instances are added to vcenter, but their default network adapter is set to disconnected, and ultimately the ansible task iteration over these two instances fail.

Understanding why this happens, I've consulted VMWare events log and /var/log/vmware-imc/toolsDeployPkg.log and my rough guesstimate based on what I can see is that there is some kind of MAC address duplication race condition taking place, where the other two guests are told to change their MAC address since they're using a duplicate, but it seems as if VMWare does not remember performing this change or what details it changed when it comes to guest instance network customization. I'm attaching my findings as screenshots, they are sorted chronologically(as vc reported back to me). The below screenshots are of one of the two faulty instances.

This isn't a one off-event, I can reproduce it 100% every single time. Is this a known issue and are there any ways to work around it? I'm currently throttling the ansible task so the VM's are created in serial, and that mitigates the issue but obviously takes quite some more time to finish. I've considered controlling the ansible loop with some delay, but it's an ugly and not quite deterministic approach to solve what I see as a vmware/open-vm-tools bug?

[PaTHml]

Hi fiskhest, Do you have some details on the versions involved VC, ESXi, open-vm-tools and which guest OS is involved.

In clone workflows, the MAC Address is updated at creation of the clone; tools are usually not involved at that point.

If you can collect the customization logs (rather than a screenshot of the logs) that would also help.

An internal bug was filed, will need info for update.

[fiskhest]

Yes, certainly!

vSphere Client version 6.7.0.40000 VMware ESXi, 6.7.0, 14320388 VMware Tools: Running, version:11269 (Guest Managed) Guest OS: Ubuntu LTS 18.04.5

I didn't save the state of the guests and have since had to move on with my assignment, but as soon as I have some breathing space I'll look into reproducing and collecting the customization logs to attach here.

[fiskhest]

Here are the toolsDeployPkg.log. The naming is a little bit different than the original issue, but the symptoms are the same.

Do note, when trying to reproduce this, I first had shorter names (still using special characters like /_-.). I was unable to reproduce in five different iterations, only when I set the guest names to longer strings did the issue occur. Could it be that the initial mac address generation somehow only hashes from the first N characters and in my scenario those N characters are all the same?

names:

miradot.service-astley-cached-redis
miradot.service-astley-cached-mysql
miradot.service-astley-cached-web

toolsDeployPkg-broken1.log toolsDeployPkg-broken2.log toolsDeployPkg-working.log

[johnwvmw]

@fiskhest Thanks for providing the requested logs. These have been attached to the internal bug report tracking this issue.

[prasadjlv]

@johnwvmw @fiskhest Was there a resolution (or workaround) for the above issue? I am running into exactly the same issue.

[johnwvmw]

Unfortunately, it appears that engineers were unable to reproduce the issue. I will contact the Guest Customization team and reopen the issue internally.

[rongz609]

@prasadjlv There is a work around method to solve this issue. You can explicitly set the MAC addresses in VirtualMachineCloneSpec(vim.vm.CloneSpec) for each newly cloned VMs to avoid the MAC competition.

When cloning a new VM from a VM template on the vCenter web UI client, you are required to fill the VirtualMachineCloneSpec, which includes Virtual Machine hardware customization parameters (stored in VirtualMachineConfigSpec). This spec is optional, you need to check the box as below to setting it: You can change the MAC address mode as "Manual", and give it your user-defined value.

And you can check the VM Clone spec API in VMODL for more information: https://vdc-repo.vmware.com/vmwb-repository/dcr-public/fa5d1ee7-fad5-4ebf-b150-bdcef1d38d35/a5e46da1-9b96-4f0c-a1d0-7b8f3ebfd4f5/doc/vim.vm.CloneSpec.html

[fiskhest]

@johnwvmw thats an extremely professional and great handling of an issue on the open tracker. When you can't reproduce internally you don't update the external case and just let it be open forever? Super unimpressed, especially since I can reproduce 100% of the time AND I gave the exact trigger that causes the issue in my environment. How about getting back to me so we can work on trying to reproduce it together? Pathetic handling from a multi billion dollar company when you get gratis help to find new bugs. I am most certainly not going to recommend VMWare to future customers.

@prasadjlv if the issue is same as mine and you have VMs like:

super_long_vm_instance_name-1
super_long_vm_instance_name-2
super_long_vm_instance_name-3

the solution for me was to:

1-super_long_vm_instance_name
2-super_long_vm_instance_name
3-super_long_vm_instance_name

[rongz609]

@fiskhest @prasadjlv Sorry for the late response and inconvenient. I'm checking whether the work around method in my last comment can work in Ansible. Will update the result later.

[fiskhest]

@rongz609 I might be remembering wrong but I'm fairly certain I tried customizing the mac address in my ansible play so each guest would get a unique mac address but it didn't matter because the boot up process first does the same procedure as in my console screenshot and then proceed to apply the new mac address. I.e. the first guest that would successfully be provisioned with the mac address I provided, but the other two guests would still be stuck pending on console and finally timing out in the ansible play.

[rongz609]

@fiskhest Thanks for your information. Could you share your Ansible playbook with me? I'm trying to reproduce it.

[rongz609]

@fiskhest @prasadjlv I tried parallel VM clone with below Ansible module: https://github.com/ansible-collections/community.vmware/blob/main/plugins/modules/vmware_guest.py I used a VM template which has no nic, and deployed three new VMs in parallel. The VM clone and guest OS customization were completed successfully, a new nic was added in each newly cloned VMs with the specified MAC addresses in my playbook. The network parameters in my playbook were as below:

For new VM 1

    networks:
      - name: 'VM Network'
        type: dhcp
        mac: '00:50:56:ab:ad:4f'
        device_type: 'vmxnet3'

For new VM 2

    networks:
      - name: 'VM Network'
        type: dhcp
        mac: '00:50:56:ab:ad:5f'
        device_type: 'vmxnet3'

For new VM 3

    networks:
      - name: 'VM Network'
        type: dhcp
        mac: '00:50:56:ab:ad:6f'
        device_type: 'vmxnet3'

I need to say that if you use other Ansible modules instead of vmware_guest.py, the playbook and test result maybe different from my test. The reason why I used a VM template with no nic is that when setting mac address in networks privilege segment, it means adding a nic to the newly cloned VM with the specified MAC address. This is determined by the python script logic in vmware_guest module. And the parameter rules of playbook can be changed by updating the python script in this module if needed.

But I failed to reproduce your issue when cloning three new VMs in parallel from a VM template which has one nic, and without setting the MAC addresses in playbook. So I'm not sure whether this method can solve your issue. In addition, if you use other Ansible module instead of vmware_guest, the issue should be solved based on the python script in that module. So please share your Ansible module and playbook with me if this issue still can't be resolved by my method.

[fiskhest]

@rongz609 thanks for that input. I currently do not have the time to reproduce your findings on my end. Here's the task I used originally:

- name: Clone a virtual machine from Linux template, ensure it is started and customize it
  vmware_guest:
    hostname: "{{ vc_hostname }}"
    username: "{{ vc_user }}"
    password: "{{ vc_pass }}"
    validate_certs: no
    datacenter: dc1
    state: poweredon
    folder: /_Templates/Linux/Labb/active
    template: "{{ vm_template }}"
    name: "{{ inventory_hostname | replace('/', '%2F') }}"
    cluster: cluster1
    hardware:
      hotadd_cpu: true
      hotadd_memory: true
      nested_virt: true
      num_cpus: "{{ req_vcpu }}"
      num_cpu_cores_per_socket: "{{ req_vcpu }}"
      memory_mb: "{{ req_mem }}"
    customization:
      hostname: "{{ inventory_hostname }}"
    networks:
    - name: "VM Network"
      device_type: vmxnet3
      type: dhcp
      start_connected: false
    - name: "{{ vm_network }}"
      device_type: vmxnet3
      type: dhcp
      start_connected: true
    wait_for_ip_address: true
    wait_for_customization: true
  delegate_to: localhost
  register: deploy

Where the inventory looked like:

miradot.service-astley-cached-redis
miradot.service-astley-cached-mysql
miradot.service-astley-cached-web

I can't find anything relevant in git history or my docs about having tested setting MAC myself, though this was a very long time ago in my world. :smile: Maybe @prasadjlv has the time and interest to verify your method faster than I, I'll try it eventually. Thanks.

[usscarter]

Just encountered this in my lab environment as well, neat.

[rongz609]

@fiskhest Thanks for your response. How do you clone multiple child VMs from one VM template in parallel? My method was using below expressions in my runbook, and "async: 3600" means to start next VM clone task without waiting for the former task finished. Then multiple child VMs clone tasks are triggered in parallel by one runbook.

async: 3600 poll:0

@usscarter @prasadjlv Would you share your method?

I'm asking this question to try to reproduce this issue in my place. Thanks.

[fiskhest]

Literal example:

# file: hosts
[these_vms_will_be_templated]
miradot.service-astley-cached-redis
miradot.service-astley-cached-mysql
miradot.service-astley-cached-web

# play.yml
---
- hosts: these_vms_will_be_templated
  connection: local
  tasks:
  - name: Clone a virtual machine from Linux template, ensure it is started and customize it
    vmware_guest:
      hostname: "{{ vc_hostname }}"
      username: "{{ vc_user }}"
      password: "{{ vc_pass }}"
      validate_certs: no
      datacenter: dc1
      state: poweredon
      folder: /_Templates/Linux/Labb/active
      template: "{{ vm_template }}"
      name: "{{ inventory_hostname | replace('/', '%2F') }}"
      cluster: cluster1
      hardware:
        hotadd_cpu: true
        hotadd_memory: true
        nested_virt: true
        num_cpus: "{{ req_vcpu }}"
        num_cpu_cores_per_socket: "{{ req_vcpu }}"
        memory_mb: "{{ req_mem }}"
      customization:
        hostname: "{{ inventory_hostname }}"
      networks:
      - name: "VM Network"
        device_type: vmxnet3
        type: dhcp
        start_connected: false
      - name: "{{ vm_network }}"
        device_type: vmxnet3
        type: dhcp
        start_connected: true
      wait_for_ip_address: true
      wait_for_customization: true
    delegate_to: localhost
    register: deploy
    # throttle: 1  # makes this task run in serial, one per host. Otherwise all three inventory_hosts are templated at roughly the same time and this bug is triggered

# execution results (this is pseudo-copied example as I don't have the time to get a working vCenter connection and waiting for the execution results)
❯ ansible-playbook -i hosts play.yml

PLAY [these_vms_will_be_templated] ***************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************
ok: [miradot.service-astley-cached-redis]
ok: [miradot.service-astley-cached-web]
ok: [miradot.service-astley-cached-mysql]

TASK [Clone a virtual machine from Linux template, ensure it is started and customize it] ********************************
changed: [miradot.service-astley-cached-web]
fatal: [miradot.service-astley-cached-mysql]: FAILED! => {...}
fatal: [miradot.service-astley-cached-redis]: FAILED! => {...}

PLAY RECAP ***************************************************************************************************************
miradot.service-astley-cached-mysql : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
miradot.service-astley-cached-redis : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
miradot.service-astley-cached-web : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 

If I'm understanding your reproduction efforts correctly, you either have one task containing the three VM definitions or three tasks containing one VM definition each running in serial (using async). I am not doing one task on one host provisioning three VMs at the same time, instead running one forked task on one host (localhost->vcenter) provisioning three VMs at the same time.

[usscarter]

I am doing the exact same thing @fiskhest is. Clone all of them at the exact same time.

[rongz609]

Thanks @fiskhest I reproduced the issue in your way. @usscarter @prasadjlv Could you use a VM template which has no nic? I found that no matter the VM template has or has no nics, the newly cloned VMs always can be configured with nics according to the "networks:" parameter in playbook. The different is: if the VM template has nics, then the existing nics will be copied and reconfigured in the newly cloned VMs. if the VM template has no nics, then the nic list from playbook will be created in the newly cloned VMs.

The MAC conflict only happens when multiple nics are copied from the same nic in parallel. So using a non-nic VM template can avoid this issue. And the mac addresses configuration is not necessary actually.

[prasadjlv]

@rongz609 @fiskhest I am sorry for the late response, I have not been following up on this thread. Our current workaround is to clone one VM at a time and as you can imagine it's painfully slow.

@rongz609 Our current process (using Ansible) is as follows:

• Upload the OVA
• Create a VM using the above OVA
• Power on the VM so that guest tools are initialized
• Shutdown the VM and convert it to a VM template
• Use the VM template to clone multiple VMs in parallel (actually clone one at a time with this bug)

If the suggestion is to remove nics from the VM template, where/how can that be done in the above process? Or Should the OVA itself have any nics?

[usscarter]

Thanks @fiskhest I reproduced the issue in your way. @usscarter @prasadjlv Could you use a VM template which has no nic? I found that no matter the VM template has or has no nics, the newly cloned VMs always can be configured with nics according to the "networks:" parameter in playbook. The different is: if the VM template has nics, then the existing nics will be copied and reconfigured in the newly cloned VMs. if the VM template has no nics, then the nic list from playbook will be created in the newly cloned VMs.

The MAC conflict only happens when multiple nics are copied from the same nic in parallel. So using a non-nic VM template can avoid this issue. And the mac addresses configuration is not necessary actually.

So,

This does appear to be working for me so far. Its a little annoying to have to packer to remove the NIC before final templating, but its not a huge deal breaker for the time this saves.

It still seems like this is an issue though, we just happen to have a workaround.

[rongz609]

I am sorry for the late response, I have not been following up on this thread. Our current workaround is to clone one VM at a time and as you can imagine it's painfully slow.

Our current process (using Ansible) is as follows:

• Upload the OVA
• Create a VM using the above OVA
• Power on the VM so that guest tools are initialized
• Shutdown the VM and convert it to a VM template
• Use the VM template to clone multiple VMs in parallel (actually clone one at a time with this bug)

If the suggestion is to remove nics from the VM template, where/how can that be done in the above process? Or Should the OVA itself have any nics?

@prasadjlv You can remove the nics after "Shutdown the VM" and before "convert it to a VM template". I think it's also OK if you use a OVA that has no nics.

[rongz609]

@prasadjlv @usscarter Another work around method is to use below statement:

async: 3600 poll:0

If you use "poll:0" to connect multiple VM clone tasks in one playbook, these tasks can also be triggered in parallel. And as I tested, the MAC conflict won't happen in this way. Maybe it's because there is tiny time delay between the tasks in this way comparing to the way in which MAC conflict appears.

See the instruction about "Run tasks concurrently: poll = 0": https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html

Yes, this is a product issue in vsphere. We will evaluate it and try to find out how to fix it. But I think it needs to take time since it's not easy to fix it.

[prasadjlv]

@rongz609 thank you for the workarounds, will try the "poll: 0" fix when time permits.

[prasadjlv]

@rongz609 the "poll: 0" trick didn't work for me (same issue as above), but removing the nics from the template worked.

[rongz609]

@prasadjlv Thanks for your update. Now we can say that only removing nics from VM template can work 100%. The "poll: 0" method can preventing MAC conflict in my environment is just an occasional case.

I used to reproduce the MAC conflict successfully, but it just disappeared after I reinstall my vCenter testbed. I spend a lot of time to restore it but still not working. Could you share me the vpxd.log when MAC conflict happens? It's /var/log/vmware/vpxd.log on the vCenter server. Thanks.

[prasadjlv]

@rongz609 I am sorry I do not have access to the vCenter to share /var/log/vmware/vpxd.log.