vMotion from one cluster/datastore to another succeeds but fails in vCenter Server with a specified parameter was not correct spec.pool

BraddMPiontek commented 4 years ago

Attempting to use Terraform to vmotion VMs from one Cluster and Datastore to another. terraform plan and terraform apply all look good. Terraform apply succeeds but the VM is not successfully relocated in VSphere. The message is "specified parameter was not correct spec.pool" Subsequent plans show the state with the VM is on the old cluster and datastore and applys attempt to move it again. (which makes sense as it failed in VCenter)

Terraform Version

Terraform v0.11.10

vSphere Provider Version

v1.16.1

Affected Resource(s)

vsphere_virtual_machine

Terraform Configuration Files

resource "vsphere_virtual_machine" "linux_dhcp_vm" {
  provider = "vsphere.module_vm"

  # Number of VMs
  # Checking vm_os_family to see if we should build that kind of VM
  count = "${var.vm_os_family == "linux" && var.static == "false" ? var.vm_count : 0}"

  # Basics
  name              = "${local.vm_host_base_name}${format("%01s",var.application_leg)}${format("%02d", count.index + 1)}"
  resource_pool_id  = "${data.vsphere_compute_cluster.cluster.resource_pool_id}"
  datastore_id      = "${data.vsphere_datastore.datastore.id}"
  folder            = "${vsphere_folder.application_vms.path}"
  annotation        = "${var.application_name}-${var.application_environment}"
  custom_attributes = "${zipmap(local.custom_attribute_keys, local.custom_attribute_vals)}"

  # CPU
  num_cpus             = "${var.vm_num_cpus}"
  num_cores_per_socket = "${var.vm_num_cores_per_socket}"
  cpu_hot_add_enabled  = true

  # Memory
  memory                 = "${var.vm_memory}"
  memory_hot_add_enabled = true

  # Storage
  scsi_type = "${data.vsphere_virtual_machine.template.scsi_type}"

  disk {
    # /dev/sda
    label       = "disk0"
    size        = "${var.vm_disk0_size == "DERIVED_DEFAULT" ? lookup(local.vm_os_family_defaults[var.vm_os_family], "disk0_size") : var.vm_disk0_size}"
    unit_number = 0
  }

  disk {
    # /dev/sdb
    label       = "disk1"
    size        = "${var.vm_disk1_size == "DERIVED_DEFAULT" ? lookup(local.vm_os_family_defaults[var.vm_os_family], "disk1_size") : var.vm_disk1_size}"
    unit_number = 1
  }

  # Network
  network_interface {
    network_id = "${data.vsphere_network.network.id}"
  }

  #
  # Clone configurations
  #
  guest_id = "${data.vsphere_virtual_machine.template.guest_id}"

  clone {
    # Longer waits for VM builds during heavy periods
    timeout = 120

    template_uuid = "${data.vsphere_virtual_machine.template.id}"

    customize {
      # This needs to be initialized to use DHCP
      network_interface {}

      linux_options {
        host_name = "${local.vm_host_base_name}${format("%01s",var.application_leg)}${format("%02d", count.index + 1)}"
        domain    = "${var.vm_domain == "DERIVED_DEFAULT" ? lookup(local.vm_domain_defaults,"${var.vm_os_family}-${var.vsphere_environment}") : var.vm_domain}"
        time_zone = "${var.vm_time_zone == "DERIVED_DEFAULT" ? lookup(local.vm_os_family_defaults[var.vm_os_family], "timezone") : var.vm_time_zone}"
      }
    }
  }
}

Debug Output

https://gist.github.com/piontekdd/0b32d7cb9cb8630bc53ce6dacad6a565.js

Panic Output

N/A

Expected Behavior

Successfully relocated the VM to the new cluster and datastore.

Actual Behavior

Terraform says the apply was successful but the VM remains on the old cluster and datastore.

Steps to Reproduce

terraform apply

Important Factoids

N/A

References

koikonom commented 4 years ago

Hi @piontekdd, thanks for taking the time to file this issue.

I noticed the VM you're trying to migrate has two disks. Out of curiosity, can you confirm you're also seeing the same issue with VMs that have a single disk?

Also would it be possible to run the the apply operation again, but this time set the TF_LOG environment variable to DEBUG (like TF_LOG=DEBUG terraform apply <planfile>) and post the debug output to another github gist?

BraddMPiontek commented 4 years ago

We do not have any VMs with single disks nor is our Terraform coded in a manner where I can easily create such a VM.

movergan commented 4 years ago

Hello,

Faced the same issue with TF12

Terraform v0.12.25
+ provider.null v2.1.2
+ provider.random v2.2.0
+ provider.vsphere v1.17.4

VM is with single disk. We use vSAN if it's important.

TF log:

Terraform will perform the following actions:

  # module.hostname1.vsphere_virtual_machine.vm_win[0] will be updated in-place
  ~ resource "vsphere_virtual_machine" "vm_win" {
        boot_delay                              = 0
        boot_retry_delay                        = 10000
        boot_retry_enabled                      = false
        change_version                          = "2020-08-21T10:01:57.12215Z"
        cpu_hot_add_enabled                     = false
        cpu_hot_remove_enabled                  = false
        cpu_limit                               = -1
        cpu_performance_counters_enabled        = false
        cpu_reservation                         = 0
        cpu_share_count                         = 2000
        cpu_share_level                         = "normal"
        custom_attributes                       = {}
      ~ datastore_id                            = "datastore-2192" -> "datastore-2156"
        default_ip_address                      = "10.10.10.10"
        efi_secure_boot_enabled                 = true
        enable_disk_uuid                        = false
        enable_logging                          = false
        ept_rvi_mode                            = "automatic"
        extra_config                            = {}
        firmware                                = "efi"
        folder                                  = "PROD"
        force_power_off                         = true
        guest_id                                = "windows9Server64Guest"
        guest_ip_addresses                      = [
            "10.10.10.10",
            "fe80::68cd:2547:93bc:f43c",
        ]
        hardware_version                        = 15
      ~ host_system_id                          = "host-2175" -> (known after apply)
        hv_mode                                 = "hvAuto"
        id                                      = "420686a2-a5d9-84e8-a897-07364d913199"
        latency_sensitivity                     = "normal"
        memory                                  = 2048
        memory_hot_add_enabled                  = false
        memory_limit                            = -1
        memory_reservation                      = 0
        memory_share_count                      = 20480
        memory_share_level                      = "normal"
        migrate_wait_timeout                    = 30
        moid                                    = "vm-2531"
        name                                    = "vmname"
        nested_hv_enabled                       = true
        num_cores_per_socket                    = 1
        num_cpus                                = 2
        poweron_timeout                         = 300
        reboot_required                         = false
      ~ resource_pool_id                        = "resgroup-2130" -> "resgroup-2128"
        run_tools_scripts_after_power_on        = true
        run_tools_scripts_after_resume          = true
        run_tools_scripts_before_guest_reboot   = false
        run_tools_scripts_before_guest_shutdown = true
        run_tools_scripts_before_guest_standby  = true
        scsi_bus_sharing                        = "noSharing"
        scsi_controller_count                   = 1
        scsi_type                               = "lsilogic-sas"
        shutdown_wait_timeout                   = 3
        storage_policy_id                       = "aa6d5a82-1c88-45da-85d3-3d74b91a5bad"
        swap_placement_policy                   = "inherit"
        sync_time_with_host                     = false
        tags                                    = []
        uuid                                    = "420686a2-a5d9-84e8-a897-07364d913199"
        vapp_transport                          = []
        vmware_tools_status                     = "guestToolsRunning"
        vmx_path                                = "2f9a3f5f-c840-f64c-6488-bc97e145fa2c/vmname.somedomain.vmx"
        wait_for_guest_ip_timeout               = 0
        wait_for_guest_net_routable             = true
        wait_for_guest_net_timeout              = 5

        clone {
            linked_clone    = false
            ovf_network_map = {}
            ovf_storage_map = {}
            template_uuid   = "4206fad3-b942-d7b0-dd29-8c97cca509f9"
            timeout         = 120

            customize {
                dns_server_list = [
                    "8.8.8.8",
                    "4.4.4.4",
                ]
                dns_suffix_list = []
                ipv4_gateway    = "10.10.10.254"
                timeout         = 40

                network_interface {
                    dns_server_list = []
                    ipv4_address    = "10.10.10.10"
                    ipv4_netmask    = 24
                    ipv6_netmask    = 0
                }

                windows_options {
                    admin_password        = (sensitive value)
                    auto_logon            = true
                    auto_logon_count      = 1
                    computer_name         = "vmname"
                    domain_admin_password = (sensitive value)
                    domain_admin_user     = "Administrator"
                    full_name             = "Administrator"
                    join_domain           = "somedomain"
                    organization_name     = "noname"
                    run_once_command_list = []
                    time_zone             = 85
                }
            }
        }

      ~ disk {
            attach            = false
          ~ datastore_id      = "datastore-2192" -> "datastore-2156"
            device_address    = "scsi:0:0"
            disk_mode         = "persistent"
            disk_sharing      = "sharingNone"
            eagerly_scrub     = false
            io_limit          = -1
            io_reservation    = 0
            io_share_count    = 1000
            io_share_level    = "normal"
            keep_on_remove    = false
            key               = 2000
            label             = "disk0"
            path              = "2f9a3f5f-c840-f64c-6488-bc97e145fa2c/vmname.vmdk"
            size              = 100
            storage_policy_id = "aa6d5a82-1c88-45da-85d3-3d74b91a5bad"
            thin_provisioned  = true
            unit_number       = 0
            uuid              = "6000C29a-b88f-e8c1-30f7-ad6ea52eeb15"
            write_through     = false
        }

        network_interface {
            adapter_type          = "e1000e"
            bandwidth_limit       = -1
            bandwidth_reservation = 0
            bandwidth_share_count = 50
            bandwidth_share_level = "normal"
            device_address        = "pci:0:7"
            key                   = 4000
            mac_address           = "00:50:56:86:46:ed"
            network_id            = "dvportgroup-472"
            use_static_mac        = false
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.vmname.vsphere_virtual_machine.vm_win[0]: Modifying... [id=420686a2-a5d9-84e8-a897-07364d913199]
module.vmname.vsphere_virtual_machine.vm_win[0]: Still modifying... [id=420686a2-a5d9-84e8-a897-07364d913199, 10s elapsed]
module.vmname.vsphere_virtual_machine.vm_win[0]: Modifications complete after 16s [id=420686a2-a5d9-84e8-a897-07364d913199]

but in fact vSphere didn't migrate the VM:

fuxjezz commented 2 years ago

With Terraform v0.14.11 , ESXi 7 and vsphere provide v2.0.2 this is still an issue....

tenthirtyam commented 2 years ago

To confirm, this issue is seen when moving between clusters, not just between datastores, correct?

Ryan Johnson Staff II Solutions Architect, VMware Inc.

sonman commented 1 year ago

Also happens in our lab. Does a workaround exists ? vSphere Provider 2.0.2

sonman commented 1 year ago

I tested again with vSphere Provider 2.4.3. I can not reproduce the issue anymore. Yay

cuongdevzinza commented 8 months ago

Hello, I'm also facing this issue when migrating vSphere VM between host

Terraform v1.6.0
vSphere provider v2.5.0

I'd tried migrating with govc client which also based on govmomi and got these results:

govc vm.migrate -host -ds vm ✅ govc vm.migrate -host -pool /Resources -ds vm ✅ govc vm.migrate -pool /Resources -ds vm ❌ Digging to terraform-vsphere-provider code and found this line: https://github.com/hashicorp/terraform-provider-vsphere/blob/main/vsphere/resource_vsphere_virtual_machine.go#L638

I have built a local provider version with the above line commented out and my vm could migrate successfully. Please take a look.

burnsjared0415 commented 2 weeks ago

Is this still a Issue? Or can it be closed based on newer Terraform version?

hashicorp / terraform-provider-vsphere