Terraform requires two destroys to clean resources and folders

[uwardlaw]

From the command line, here is an example of destroying resources for student01.

control@control:~/rous/terraform$ terraform workspace select student01
Switched to workspace "student01".
control@control:~/rous/terraform$ terraform -chdir=/home/control/rous/terraform destroy -auto-ap                                                                                                  prove -var="port_group_name1=fundamentalsEnclave_class21012_student01_net00" -var="vsphere_virtu                                                                                                  al_machine_path=class21012/student01" -var="switch_name_1=class21012_student01_switch00" -var="p                                                                                                  arent_folder=class21012" -var="student_folder=student01"
vsphere_folder.parent: Refreshing state... [id=group-v820]
vsphere_host_virtual_switch.hvs1: Refreshing state... [id=tf-HostVirtualSwitch:host-64:class2101                                                                                                  2_student01_switch00]
vsphere_host_port_group.pg1: Refreshing state... [id=tf-HostPortGroup:host-64:fundamentalsEnclav                                                                                                  e_class21012_student01_net00]
time_sleep.wait_on_net: Refreshing state... [id=2021-11-16T20:59:38Z]
vsphere_virtual_machine.cloned_virtual_machine1: Refreshing state... [id=423df244-1006-9fa4-4651                                                                                                  -6882e39c706b]
vsphere_folder.folder: Refreshing state... [id=group-v821]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform
apply":

  # vsphere_virtual_machine.cloned_virtual_machine1 has been changed
  ~ resource "vsphere_virtual_machine" "cloned_virtual_machine1" {
      + custom_attributes                       = {}
      + extra_config                            = {}
        id                                      = "423df244-1006-9fa4-4651-6882e39c706b"
        name                                    = "win10_domain_clone"
      + tags                                    = []
        # (57 unchanged attributes hidden)

      ~ clone {
            # (3 unchanged attributes hidden)

          ~ customize {
              + dns_server_list = []
              + dns_suffix_list = []
                # (2 unchanged attributes hidden)

              ~ network_interface {
                  + dns_server_list = []
                    # (3 unchanged attributes hidden)
                }

              ~ windows_options {
                  + run_once_command_list = []
                    # (8 unchanged attributes hidden)
                }
            }
        }

        # (2 unchanged blocks hidden)
    }
  # vsphere_folder.parent has been changed
  ~ resource "vsphere_folder" "parent" {
      + custom_attributes = {}
        id                = "group-v820"
      + tags              = []
        # (3 unchanged attributes hidden)
    }
  # vsphere_folder.folder has been changed
  ~ resource "vsphere_folder" "folder" {
      + custom_attributes = {}
        id                = "group-v821"
      + tags              = []
        # (3 unchanged attributes hidden)
    }
  # vsphere_host_port_group.pg1 has been changed
  ~ resource "vsphere_host_port_group" "pg1" {
        id                        = "tf-HostPortGroup:host-64:fundamentalsEnclave_class21012_stu                                                                                                  dent01_net00"
        name                      = "fundamentalsEnclave_class21012_student01_net00"
      ~ ports                     = [
          + {
              + key           = "key-vim.host.PortGroup.Port-201342978"
              + mac_addresses = [
                  + "00:50:56:bd:ee:df",
                ]
              + type          = "virtualMachine"
            },
        ]
        # (8 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant
attributes using ignore_changes, the following plan may include actions to undo or respond to
these changes.

───────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # time_sleep.wait_on_net will be destroyed
  - resource "time_sleep" "wait_on_net" {
      - create_duration = "10s" -> null
      - id              = "2021-11-16T20:59:38Z" -> null
    }

  # vsphere_folder.folder will be destroyed
  - resource "vsphere_folder" "folder" {
      - custom_attributes = {} -> null
      - datacenter_id     = "datacenter-21" -> null
      - id                = "group-v821" -> null
      - path              = "class21012/student01" -> null
      - tags              = [] -> null
      - type              = "vm" -> null
    }

  # vsphere_folder.parent will be destroyed
  - resource "vsphere_folder" "parent" {
      - custom_attributes = {} -> null
      - datacenter_id     = "datacenter-21" -> null
      - id                = "group-v820" -> null
      - path              = "class21012" -> null
      - tags              = [] -> null
      - type              = "vm" -> null
    }

  # vsphere_host_port_group.pg1 will be destroyed
  - resource "vsphere_host_port_group" "pg1" {
      - computed_policy           = {
          - "allow_forged_transmits"    = "true"
          - "allow_mac_changes"         = "true"
          - "allow_promiscuous"         = "false"
          - "check_beacon"              = "false"
          - "failback"                  = "true"
          - "notify_switches"           = "true"
          - "shaping_average_bandwidth" = "0"
          - "shaping_burst_size"        = "0"
          - "shaping_enabled"           = "false"
          - "shaping_peak_bandwidth"    = "0"
          - "teaming_policy"            = "loadbalance_srcid"
        } -> null
      - host_system_id            = "host-64" -> null
      - id                        = "tf-HostPortGroup:host-64:fundamentalsEnclave_class21012_stu                                                                                                  dent01_net00" -> null
      - key                       = "key-vim.host.PortGroup-fundamentalsEnclave_class21012_stude                                                                                                  nt01_net00" -> null
      - name                      = "fundamentalsEnclave_class21012_student01_net00" -> null
      - ports                     = [
          - {
              - key           = "key-vim.host.PortGroup.Port-201342978"
              - mac_addresses = [
                  - "00:50:56:bd:ee:df",
                ]
              - type          = "virtualMachine"
            },
        ] -> null
      - shaping_average_bandwidth = 0 -> null
      - shaping_burst_size        = 0 -> null
      - shaping_peak_bandwidth    = 0 -> null
      - virtual_switch_name       = "class21012_student01_switch00" -> null
      - vlan_id                   = 0 -> null
    }

  # vsphere_host_virtual_switch.hvs1 will be destroyed
  - resource "vsphere_host_virtual_switch" "hvs1" {
      - active_nics               = [] -> null
      - allow_forged_transmits    = true -> null
      - allow_mac_changes         = true -> null
      - allow_promiscuous         = false -> null
      - beacon_interval           = 1 -> null
      - check_beacon              = false -> null
      - failback                  = true -> null
      - host_system_id            = "host-64" -> null
      - id                        = "tf-HostVirtualSwitch:host-64:class21012_student01_switch00"                                                                                                   -> null
      - link_discovery_operation  = "listen" -> null
      - link_discovery_protocol   = "cdp" -> null
      - mtu                       = 1500 -> null
      - name                      = "class21012_student01_switch00" -> null
      - network_adapters          = [] -> null
      - notify_switches           = true -> null
      - number_of_ports           = 128 -> null
      - shaping_average_bandwidth = 0 -> null
      - shaping_burst_size        = 0 -> null
      - shaping_enabled           = false -> null
      - shaping_peak_bandwidth    = 0 -> null
      - standby_nics              = [] -> null
      - teaming_policy            = "loadbalance_srcid" -> null
    }

  # vsphere_virtual_machine.cloned_virtual_machine1 will be destroyed
  - resource "vsphere_virtual_machine" "cloned_virtual_machine1" {
      - annotation                              = <<-EOT
            Leia/princess
            Administrator/P@ssword123
        EOT -> null
      - boot_delay                              = 0 -> null
      - boot_retry_delay                        = 10000 -> null
      - boot_retry_enabled                      = false -> null
      - change_version                          = "2021-11-16T21:00:23.05224Z" -> null
      - cpu_hot_add_enabled                     = false -> null
      - cpu_hot_remove_enabled                  = false -> null
      - cpu_limit                               = -1 -> null
      - cpu_performance_counters_enabled        = false -> null
      - cpu_reservation                         = 0 -> null
      - cpu_share_count                         = 1000 -> null
      - cpu_share_level                         = "normal" -> null
      - custom_attributes                       = {} -> null
      - datastore_id                            = "datastore-65" -> null
      - default_ip_address                      = "10.0.0.10" -> null
      - efi_secure_boot_enabled                 = false -> null
      - enable_disk_uuid                        = false -> null
      - enable_logging                          = false -> null
      - ept_rvi_mode                            = "automatic" -> null
      - extra_config                            = {} -> null
      - firmware                                = "efi" -> null
      - folder                                  = "class21012/student01" -> null
      - force_power_off                         = true -> null
      - guest_id                                = "windows9_64Guest" -> null
      - guest_ip_addresses                      = [
          - "10.0.0.10",
          - "fe80::20ef:e28c:339f:269",
        ] -> null
      - host_system_id                          = "host-64" -> null
      - hv_mode                                 = "hvAuto" -> null
      - id                                      = "423df244-1006-9fa4-4651-6882e39c706b" -> null
      - latency_sensitivity                     = "normal" -> null
      - memory                                  = 1024 -> null
      - memory_hot_add_enabled                  = false -> null
      - memory_limit                            = -1 -> null
      - memory_reservation                      = 0 -> null
      - memory_share_count                      = 10240 -> null
      - memory_share_level                      = "normal" -> null
      - migrate_wait_timeout                    = 30 -> null
      - moid                                    = "vm-823" -> null
      - name                                    = "win10_domain_clone" -> null
      - nested_hv_enabled                       = false -> null
      - num_cores_per_socket                    = 1 -> null
      - num_cpus                                = 1 -> null
      - reboot_required                         = false -> null
      - resource_pool_id                        = "resgroup-89" -> null
      - run_tools_scripts_after_power_on        = true -> null
      - run_tools_scripts_after_resume          = true -> null
      - run_tools_scripts_before_guest_reboot   = false -> null
      - run_tools_scripts_before_guest_shutdown = true -> null
      - run_tools_scripts_before_guest_standby  = true -> null
      - scsi_bus_sharing                        = "noSharing" -> null
      - scsi_controller_count                   = 1 -> null
      - scsi_type                               = "lsilogic-sas" -> null
      - shutdown_wait_timeout                   = 3 -> null
      - swap_placement_policy                   = "inherit" -> null
      - sync_time_with_host                     = false -> null
      - tags                                    = [] -> null
      - uuid                                    = "423df244-1006-9fa4-4651-6882e39c706b" -> null
      - vapp_transport                          = [] -> null
      - vmware_tools_status                     = "guestToolsRunning" -> null
      - vmx_path                                = "win10_domain_clone/win10_domain_clone.vmx" ->                                                                                                   null
      - wait_for_guest_ip_timeout               = 0 -> null
      - wait_for_guest_net_routable             = false -> null
      - wait_for_guest_net_timeout              = 1 -> null

      - clone {
          - linked_clone  = false -> null
          - template_uuid = "423db02b-ed48-d4f4-f041-18391ac57d74" -> null
          - timeout       = 30 -> null

          - customize {
              - dns_server_list = [] -> null
              - dns_suffix_list = [] -> null
              - ipv4_gateway    = "10.0.0.1" -> null
              - timeout         = 10 -> null

              - network_interface {
                  - dns_server_list = [] -> null
                  - ipv4_address    = "10.0.0.10" -> null
                  - ipv4_netmask    = 24 -> null
                  - ipv6_netmask    = 0 -> null
                }

              - windows_options {
                  - admin_password        = (sensitive value)
                  - auto_logon            = true -> null
                  - auto_logon_count      = 1 -> null
                  - computer_name         = "test1" -> null
                  - full_name             = "Administrator" -> null
                  - organization_name     = "Managed by Terraform" -> null
                  - run_once_command_list = [] -> null
                  - time_zone             = 85 -> null
                  - workgroup             = "WORKGROUP" -> null
                }
            }
        }

      - disk {
          - attach           = false -> null
          - datastore_id     = "datastore-65" -> null
          - device_address   = "scsi:0:0" -> null
          - disk_mode        = "persistent" -> null
          - disk_sharing     = "sharingNone" -> null
          - eagerly_scrub    = false -> null
          - io_limit         = -1 -> null
          - io_reservation   = 0 -> null
          - io_share_count   = 1000 -> null
          - io_share_level   = "normal" -> null
          - keep_on_remove   = false -> null
          - key              = 2000 -> null
          - label            = "disk0" -> null
          - path             = "win10_domain_clone/win10_domain_clone.vmdk" -> null
          - size             = 32 -> null
          - thin_provisioned = false -> null
          - unit_number      = 0 -> null
          - uuid             = "6000C290-67b2-62d5-f9da-11b0991a927f" -> null
          - write_through    = false -> null
        }

      - network_interface {
          - adapter_type          = "vmxnet3" -> null
          - bandwidth_limit       = -1 -> null
          - bandwidth_reservation = 0 -> null
          - bandwidth_share_count = 50 -> null
          - bandwidth_share_level = "normal" -> null
          - device_address        = "pci:0:7" -> null
          - key                   = 4000 -> null
          - mac_address           = "00:50:56:bd:ee:df" -> null
          - network_id            = "network-822" -> null
          - use_static_mac        = false -> null
        }
    }

Plan: 0 to add, 0 to change, 6 to destroy.
vsphere_folder.folder: Destroying... [id=group-v821]
vsphere_virtual_machine.cloned_virtual_machine1: Destroying... [id=423df244-1006-9fa4-4651-6882e                                                                                                  39c706b]
vsphere_virtual_machine.cloned_virtual_machine1: Still destroying... [id=423df244-1006-9fa4-4651                                                                                                  -6882e39c706b, 10s elapsed]
vsphere_virtual_machine.cloned_virtual_machine1: Destruction complete after 18s
time_sleep.wait_on_net: Destroying... [id=2021-11-16T20:59:38Z]
time_sleep.wait_on_net: Destruction complete after 0s
vsphere_host_port_group.pg1: Destroying... [id=tf-HostPortGroup:host-64:fundamentalsEnclave_clas                                                                                                  s21012_student01_net00]
vsphere_host_port_group.pg1: Destruction complete after 0s
vsphere_host_virtual_switch.hvs1: Destroying... [id=tf-HostVirtualSwitch:host-64:class21012_stud                                                                                                  ent01_switch00]
vsphere_host_virtual_switch.hvs1: Destruction complete after 0s
╷
│ Warning: Version constraints inside provider configuration blocks are deprecated
│
│   on main.tf line 16, in provider "vsphere":
│   16:   version = "<1.16.0"
│
│ Terraform 0.13 and earlier allowed provider version constraints inside the provider
│ configuration block, but that is now deprecated and will be removed in a future version of
│ Terraform. To silence this warning, move the provider version constraint into the
│ required_providers block.
╵
╷
│ Error: folder is not empty, please remove all items before deleting
│
│
╵
control@control:~/rous/terraform$ terraform -chdir=/home/control/rous/terraform destroy -auto-ap                                                                                                  prove -var="port_group_name1=fundamentalsEnclave_class21012_student01_net00" -var="vsphere_virtu                                                                                                  al_machine_path=class21012/student01" -var="switch_name_1=class21012_student01_switch00" -var="p                                                                                                  arent_folder=class21012" -var="student_folder=student01"
vsphere_folder.parent: Refreshing state... [id=group-v820]
vsphere_folder.folder: Refreshing state... [id=group-v821]

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # vsphere_folder.folder will be destroyed
  - resource "vsphere_folder" "folder" {
      - custom_attributes = {} -> null
      - datacenter_id     = "datacenter-21" -> null
      - id                = "group-v821" -> null
      - path              = "class21012/student01" -> null
      - tags              = [] -> null
      - type              = "vm" -> null
    }

  # vsphere_folder.parent will be destroyed
  - resource "vsphere_folder" "parent" {
      - custom_attributes = {} -> null
      - datacenter_id     = "datacenter-21" -> null
      - id                = "group-v820" -> null
      - path              = "class21012" -> null
      - tags              = [] -> null
      - type              = "vm" -> null
    }

Plan: 0 to add, 0 to change, 2 to destroy.
vsphere_folder.folder: Destroying... [id=group-v821]
vsphere_folder.folder: Destruction complete after 0s
vsphere_folder.parent: Destroying... [id=group-v820]
vsphere_folder.parent: Destruction complete after 0s
╷
│ Warning: Version constraints inside provider configuration blocks are deprecated
│
│   on main.tf line 16, in provider "vsphere":
│   16:   version = "<1.16.0"
│
│ Terraform 0.13 and earlier allowed provider version constraints inside the provider
│ configuration block, but that is now deprecated and will be removed in a future version of
│ Terraform. To silence this warning, move the provider version constraint into the
│ required_providers block.
╵

Destroy complete! Resources: 2 destroyed.

In terraformDestroyEnclave.yml we add ignore_errors to the first terraform destroy that removes vm resources and then use a second terraform destroy that fails on errors to clean up.

[marissaeinhorn]

At first glance this appears to be a known problem

• race condition on destroy
• destroy doesnt use dependency info