`vsphere_host_virtual_switch` resource attempts to change the VMNIC order

erSitzt commented 1 year ago

Community Guidelines

[X] I have read and agree to the HashiCorp Community Guidelines .
[X] Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
[X] Do not leave "+1" or other comments that do not add relevant information or questions.
[X] If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Terraform

v1.5.6

Terraform Provider

v2.4.3

VMware vSphere

v7.0.3

Description

Same problem as in #1503

Current vmnic order is

But terraform wants to update the order on multiple servers.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.vswitch["esxi-4.json"].vsphere_host_virtual_switch.iscsi-switch will be updated in-place
  ~ resource "vsphere_host_virtual_switch" "iscsi-switch" {
        id                        = "tf-HostVirtualSwitch:host-643727:vSwitch-iSCSI"
        name                      = "vSwitch-iSCSI"
      ~ network_adapters          = [
          - "vmnic2",
            "vmnic0",
          + "vmnic2",
        ]
        # (19 unchanged attributes hidden)
    }

  # module.vswitch["esxi-5.json"].vsphere_host_virtual_switch.iscsi-switch will be updated in-place
  ~ resource "vsphere_host_virtual_switch" "iscsi-switch" {
        id                        = "tf-HostVirtualSwitch:host-637250:vSwitch-iSCSI"
        name                      = "vSwitch-iSCSI"
      ~ network_adapters          = [
          - "vmnic2",
            "vmnic0",
          + "vmnic2",
        ]
        # (19 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Fun fact: I have one server where the vmnic order is the other way round

...and terraform is not trying to reorder the vmnics on this host

To be clear, the current vmnic order on all of those hosts currently matches what is configured in terraform.

Affected Resources or Data Sources

resource/vsphere_host_virtual_switch

Terraform Configuration

Nothing special here i think...

https://gist.github.com/erSitzt/6f7176babcefe6dab8464a3e48e9cda5

The json used here looks like this

{
    "esx-nummer"                :   4,
    "dnsname"                   :   "esxi-4.domain.com",
    "configname"                :   "esxi-4.json",
    "iscsi_1-ip"                :   "192.168.101.204",
    "iscsi_1-nic"               :   "vmnic0",
    "iscsi_2-ip"                :   "192.168.102.204",
    "iscsi_2-nic"               :   "vmnic2"
}

Debug Output

https://gist.github.com/erSitzt/bd4661e41f42c054516ff8decfe7449d

Panic Output

No response

Expected Behavior

No change detected..

Actual Behavior

Change detected on every plan/apply run

Steps to Reproduce

Not sure, will add info if i can test more

Environment Details

No response

Screenshots

No response

References

No response

github-actions[bot] commented 1 year ago

Hello, erSitzt! 🖐

Thank you for submitting an issue for this provider. The issue will now enter into the issue lifecycle.

If you want to contribute to this project, please review the contributing guidelines and information on submitting pull requests.

erSitzt commented 1 year ago

quick note, i switched cables on the host , where the order was different`and adjusted the json in my terraform project, now terraform detects a change on this host on every run as well.

So it might have something to do with vmnic0 being first in the list ?

spacegospod commented 4 months ago

Hey @erSitzt the only way I was able to reproduce this was to change the failover order outside TF and attempt to plan/apply the original configuration again.

There is no special handling in the provider in terms of adapter ordering and there's definitely nothing special about vmnic0. Can you confirm that nobody and nothing apart from Terraform is making changes to this host?

erSitzt commented 4 months ago

i will recheck this tomorrow, but im pretty sure this was a change that terraform would want to do on every run, even if they were only seconds apart with no other process/user changing any settings in vmware.

...but let be verify if this is still happening with current versions...

erSitzt commented 4 months ago

@spacegospod Still happening with current version v2.8.2

It still happens for all hosts after some time.. not sure what is triggering it. BUT i can provoke this bei entering and exiting maintenance mode for a host. Tried this for 3 out of 8 hosts and all 3 showed up in terraform plan after cycling maintenance mode.

spacegospod commented 4 months ago

@spacegospod Still happening with current version v2.8.2

It still happens for all hosts after some time.. not sure what is triggering it. BUT i can provoke this bei entering and exiting maintenance mode for a host. Tried this for 3 out of 8 hosts and all 3 showed up in terraform plan after cycling maintenance mode.

Thanks for re-testing. Is the order of the active adapters messed up in the vCenter UI too?

erSitzt commented 4 months ago

Nope vCenter is always in the same order. I noticed that after some time all hosts show up with terraform plan, but not all at the same time. We have a very basic VMware setup, no 3rd party integrations used to configure stuff and no other admins changing stuff while this is happening.

spacegospod commented 4 months ago

Looks like entering maintenance mode was the missing piece of the puzzle. I was able to reproduce the bug in 2 out of 6 attempts, it doesn't happen every time on my setup.

I noticed that after exiting maintenance mode the adapters are flipped on the HostVirtualSwitchBondBridge but are left intact on the HostNetworkPolicy. I don't know why the terraform provider uses the bridge property but it is definitely wrong. It should rely on the network policy whenever ordering is of importance.

References: https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/8.0.3/data-structures/HostVirtualSwitchBondBridge https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/8.0.3/data-structures/HostNicOrderPolicy

erSitzt commented 4 months ago

@spacegospod nice :) thanks for investigating

Funny thing why this is happening for all hosts after some time.. first i suspected it had something to do with ha primary election, because that was the first host i sent into maintenance.

hashicorp / terraform-provider-vsphere