Creating vapp_entity fails with: "The object 'vim.VirtualMachine:resgroup-v415598' has already been deleted or has not been completely created"

erSitzt commented 3 years ago

Terraform Version

v0.14.5

vSphere Provider Version

v1.24.3

Affected Resource(s)

vsphere_vapp_entity

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this. --->

Terraform Configuration Files

resource "vsphere_vapp_container" "vapp_container_cluster" {
  name                    = var.cluster_name
  parent_folder_id        = data.vsphere_folder.folder.id
  parent_resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
}

resource "vsphere_vapp_container" "vapp_container_controlplane" {
  name                    = "${var.cluster_name}-controlplane"
  parent_folder_id        = vsphere_vapp_container.vapp_container_cluster.id
  parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}

resource "vsphere_vapp_container" "vapp_container_worker" {
  name                    = "${var.cluster_name}-worker"
  parent_folder_id        = vsphere_vapp_container.vapp_container_cluster.id
  parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id
}

resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {

  target_id    = vsphere_vapp_container.vapp_container_controlplane.id
  container_id = vsphere_vapp_container.vapp_container_cluster.id
  start_order = 1
  start_action = "powerOn"
  stop_action = "guestShutdown"
}

resource "vsphere_vapp_entity" "vapp_entity_vappworker" {

  target_id    = vsphere_vapp_container.vapp_container_worker.id
  container_id = vsphere_vapp_container.vapp_container_cluster.id
  start_order = 2
  start_action = "powerOn"
  stop_action = "guestShutdown"
}

I have ommitted the other vsphere resources as they are all default stuff... i can add them if necessary :)

Debug Output

I'll add debug output after cleaning up all sensitive data

Panic Output

Expected Behavior

Settings for nested vApps should have been updated

Actual Behavior

Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415598' has already been deleted or has not been completely created

Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v415599' has already been deleted or has not been completely created

Steps to Reproduce

created a nested vApp config like above with settings for

start_order
start_action
stop_action

Important Factoids

References

0000

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

erSitzt commented 3 years ago

Another thing i noticed.. sometimes if there were like 3 vms in a vApp, the settings where applied to one of them in the first run and to the others in the second run of terraform

erSitzt commented 3 years ago

Still a problem with

❯ terraform version
Terraform v0.15.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.1.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/vsphere v1.26.0
+ provider registry.terraform.io/rancher/rke v1.0.1

Can anyone verify the problems with nested vApps ?

tenthirtyam commented 2 years ago

Are you still seeing issue or was it resolved since originally posted?

Ryan

erSitzt commented 2 years ago

I'll try to reproduce this and post an update or close the issue

tenthirtyam commented 2 years ago

Thanks for the update @erSitzt. Happy to help if it's still a concern.

Ryan

erSitzt commented 2 years ago

Seems this is still not working...

❯ terraform version
Terraform v1.1.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/vsphere v2.0.2

On first run it fails with there errors:

vsphere_vapp_container.vapp_container_cluster: Creating...
vsphere_vapp_container.vapp_container_cluster: Creation complete after 1s [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creating...
vsphere_vapp_container.vapp_container_controlplane: Creation complete after 0s [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Creating...
vsphere_vapp_container.vapp_container_worker: Creation complete after 1s [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappworker: Creating...
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│   with vsphere_vapp_entity.vapp_entity_vappcontrolplane,
│   on main.tf line 40, in resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane":
│   40: resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│   with vsphere_vapp_entity.vapp_entity_vappworker,
│   on main.tf line 49, in resource "vsphere_vapp_entity" "vapp_entity_vappworker":
│   49: resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
│ 
╵

Running apply again, does not fix it

❯ terraform apply
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # vsphere_vapp_container.vapp_container_cluster has changed
  ~ resource "vsphere_vapp_container" "vapp_container_cluster" {
        id                      = "resgroup-v522179"
        name                    = "test-vapp-issue"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_container.vapp_container_controlplane has changed
  ~ resource "vsphere_vapp_container" "vapp_container_controlplane" {
        id                      = "resgroup-v522180"
        name                    = "test-controlplane"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_container.vapp_container_worker has changed
  ~ resource "vsphere_vapp_container" "vapp_container_worker" {
        id                      = "resgroup-v522181"
        name                    = "test-worker"
      + tags                    = []
        # (12 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane has changed
  ~ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
        id             = "resgroup-v522179:resgroup-v522180"
      ~ stop_action    = "guestShutdown" -> "powerOff"
        # (7 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappworker has changed
  ~ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
        id             = "resgroup-v522179:resgroup-v522181"
      ~ stop_action    = "guestShutdown" -> "powerOff"
        # (7 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
      ~ id             = "resgroup-v522179:resgroup-v522180" -> (known after apply)
      ~ stop_action    = "powerOff" -> "guestShutdown"
        # (7 unchanged attributes hidden)
    }

  # vsphere_vapp_entity.vapp_entity_vappworker is tainted, so must be replaced
-/+ resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
      ~ id             = "resgroup-v522179:resgroup-v522181" -> (known after apply)
      ~ stop_action    = "powerOff" -> "guestShutdown"
        # (7 unchanged attributes hidden)
    }

Plan: 2 to add, 0 to change, 2 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│ 
╵

And destroy will fail too...

❯ terraform destroy
vsphere_vapp_container.vapp_container_cluster: Refreshing state... [id=resgroup-v522179]
vsphere_vapp_container.vapp_container_worker: Refreshing state... [id=resgroup-v522181]
vsphere_vapp_container.vapp_container_controlplane: Refreshing state... [id=resgroup-v522180]
vsphere_vapp_entity.vapp_entity_vappworker: Refreshing state... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Refreshing state... [id=resgroup-v522179:resgroup-v522180]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # vsphere_vapp_container.vapp_container_cluster will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_cluster" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522179" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-vapp-issue" -> null
      - parent_folder_id        = "group-v3" -> null
      - parent_resource_pool_id = "resgroup-8" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_container.vapp_container_controlplane will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_controlplane" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522180" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-controlplane" -> null
      - parent_folder_id        = "resgroup-v522179" -> null
      - parent_resource_pool_id = "resgroup-v522179" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_container.vapp_container_worker will be destroyed
  - resource "vsphere_vapp_container" "vapp_container_worker" {
      - cpu_expandable          = true -> null
      - cpu_limit               = -1 -> null
      - cpu_reservation         = 0 -> null
      - cpu_share_level         = "normal" -> null
      - cpu_shares              = 4000 -> null
      - id                      = "resgroup-v522181" -> null
      - memory_expandable       = true -> null
      - memory_limit            = -1 -> null
      - memory_reservation      = 0 -> null
      - memory_share_level      = "normal" -> null
      - memory_shares           = 163840 -> null
      - name                    = "test-worker" -> null
      - parent_folder_id        = "resgroup-v522179" -> null
      - parent_resource_pool_id = "resgroup-v522179" -> null
      - tags                    = [] -> null
    }

  # vsphere_vapp_entity.vapp_entity_vappcontrolplane will be destroyed
  - resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" {
      - container_id   = "resgroup-v522179" -> null
      - id             = "resgroup-v522179:resgroup-v522180" -> null
      - start_action   = "powerOn" -> null
      - start_delay    = 120 -> null
      - start_order    = 1 -> null
      - stop_action    = "powerOff" -> null
      - stop_delay     = 120 -> null
      - target_id      = "resgroup-v522180" -> null
      - wait_for_guest = false -> null
    }

  # vsphere_vapp_entity.vapp_entity_vappworker will be destroyed
  - resource "vsphere_vapp_entity" "vapp_entity_vappworker" {
      - container_id   = "resgroup-v522179" -> null
      - id             = "resgroup-v522179:resgroup-v522181" -> null
      - start_action   = "powerOn" -> null
      - start_delay    = 120 -> null
      - start_order    = 2 -> null
      - stop_action    = "powerOff" -> null
      - stop_delay     = 120 -> null
      - target_id      = "resgroup-v522181" -> null
      - wait_for_guest = false -> null
    }

Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

vsphere_vapp_entity.vapp_entity_vappworker: Destroying... [id=resgroup-v522179:resgroup-v522181]
vsphere_vapp_entity.vapp_entity_vappcontrolplane: Destroying... [id=resgroup-v522179:resgroup-v522180]
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522181' has already been deleted or has not been completely created
│ 
│ 
╵
╷
│ Error: ServerFaultCode: The object 'vim.VirtualMachine:resgroup-v522180' has already been deleted or has not been completely created
│ 
│ 
╵

erSitzt commented 2 years ago

Btw this is the resulting vApp

erSitzt commented 2 years ago

And this the state file after all this

terraform.tfstate

```json { "version": 4, "terraform_version": "1.1.5", "serial": 9, "lineage": "4f7eef30-b7a8-8da2-31ea-7375f11405c7", "outputs": {}, "resources": [ { "mode": "data", "type": "vsphere_compute_cluster", "name": "cluster", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "datacenter_id": "datacenter-2", "id": "domain-c7", "name": "Cluster1", "resource_pool_id": "resgroup-8" }, "sensitive_attributes": [] } ] }, { "mode": "data", "type": "vsphere_datacenter", "name": "dc", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "id": "datacenter-2", "name": "DC1" }, "sensitive_attributes": [] } ] }, { "mode": "data", "type": "vsphere_folder", "name": "folder", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "id": "group-v3", "path": "/DC1/vm" }, "sensitive_attributes": [] } ] }, { "mode": "managed", "type": "vsphere_vapp_container", "name": "vapp_container_cluster", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "cpu_expandable": true, "cpu_limit": -1, "cpu_reservation": 0, "cpu_share_level": "normal", "cpu_shares": 4000, "custom_attributes": null, "id": "resgroup-v522179", "memory_expandable": true, "memory_limit": -1, "memory_reservation": 0, "memory_share_level": "normal", "memory_shares": 163840, "name": "test-vapp-issue", "parent_folder_id": "group-v3", "parent_resource_pool_id": "resgroup-8", "tags": [] }, "sensitive_attributes": [], "private": "bnVsbA==", "dependencies": [ "data.vsphere_compute_cluster.cluster", "data.vsphere_datacenter.dc", "data.vsphere_folder.folder" ] } ] }, { "mode": "managed", "type": "vsphere_vapp_container", "name": "vapp_container_controlplane", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "cpu_expandable": true, "cpu_limit": -1, "cpu_reservation": 0, "cpu_share_level": "normal", "cpu_shares": 4000, "custom_attributes": null, "id": "resgroup-v522180", "memory_expandable": true, "memory_limit": -1, "memory_reservation": 0, "memory_share_level": "normal", "memory_shares": 163840, "name": "test-controlplane", "parent_folder_id": "resgroup-v522179", "parent_resource_pool_id": "resgroup-v522179", "tags": [] }, "sensitive_attributes": [], "private": "bnVsbA==", "dependencies": [ "data.vsphere_compute_cluster.cluster", "data.vsphere_datacenter.dc", "data.vsphere_folder.folder", "vsphere_vapp_container.vapp_container_cluster" ] } ] }, { "mode": "managed", "type": "vsphere_vapp_container", "name": "vapp_container_worker", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "schema_version": 0, "attributes": { "cpu_expandable": true, "cpu_limit": -1, "cpu_reservation": 0, "cpu_share_level": "normal", "cpu_shares": 4000, "custom_attributes": null, "id": "resgroup-v522181", "memory_expandable": true, "memory_limit": -1, "memory_reservation": 0, "memory_share_level": "normal", "memory_shares": 163840, "name": "test-worker", "parent_folder_id": "resgroup-v522179", "parent_resource_pool_id": "resgroup-v522179", "tags": [] }, "sensitive_attributes": [], "private": "bnVsbA==", "dependencies": [ "data.vsphere_compute_cluster.cluster", "data.vsphere_datacenter.dc", "data.vsphere_folder.folder", "vsphere_vapp_container.vapp_container_cluster" ] } ] }, { "mode": "managed", "type": "vsphere_vapp_entity", "name": "vapp_entity_vappcontrolplane", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "status": "tainted", "schema_version": 0, "attributes": { "container_id": "resgroup-v522179", "custom_attributes": null, "id": "resgroup-v522179:resgroup-v522180", "start_action": "powerOn", "start_delay": 120, "start_order": 1, "stop_action": "powerOff", "stop_delay": 120, "tags": null, "target_id": "resgroup-v522180", "wait_for_guest": false }, "sensitive_attributes": [], "private": "bnVsbA==", "dependencies": [ "data.vsphere_compute_cluster.cluster", "data.vsphere_datacenter.dc", "data.vsphere_folder.folder", "vsphere_vapp_container.vapp_container_cluster", "vsphere_vapp_container.vapp_container_controlplane" ] } ] }, { "mode": "managed", "type": "vsphere_vapp_entity", "name": "vapp_entity_vappworker", "provider": "provider[\"registry.terraform.io/hashicorp/vsphere\"]", "instances": [ { "status": "tainted", "schema_version": 0, "attributes": { "container_id": "resgroup-v522179", "custom_attributes": null, "id": "resgroup-v522179:resgroup-v522181", "start_action": "powerOn", "start_delay": 120, "start_order": 2, "stop_action": "powerOff", "stop_delay": 120, "tags": null, "target_id": "resgroup-v522181", "wait_for_guest": false }, "sensitive_attributes": [], "private": "bnVsbA==", "dependencies": [ "data.vsphere_compute_cluster.cluster", "data.vsphere_datacenter.dc", "data.vsphere_folder.folder", "vsphere_vapp_container.vapp_container_cluster", "vsphere_vapp_container.vapp_container_worker" ] } ] } ] } ```

tenthirtyam commented 2 years ago

Hi @erSitzt,

Would you be able to share your example configuration files?

Ryan Johnson Staff II Solutions Architect | VMware, Inc.

erSitzt commented 2 years ago

Sure this is my test file

Click to expand!

``` provider "vsphere" { user = "administrator@vsphere.local" password = "mypass" vsphere_server = "vcenter.domain" allow_unverified_ssl = true } data "vsphere_datacenter" "dc" { name = "DC1" } data "vsphere_compute_cluster" "cluster" { name = "Cluster1" datacenter_id = data.vsphere_datacenter.dc.id } data "vsphere_folder" "folder" { path = "/DC1/vm" } resource "vsphere_vapp_container" "vapp_container_cluster" { name = "test-vapp-issue" parent_folder_id = data.vsphere_folder.folder.id parent_resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id } resource "vsphere_vapp_container" "vapp_container_controlplane" { name = "test-controlplane" parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id } resource "vsphere_vapp_container" "vapp_container_worker" { name = "test-worker" parent_folder_id = vsphere_vapp_container.vapp_container_cluster.id parent_resource_pool_id = vsphere_vapp_container.vapp_container_cluster.id } resource "vsphere_vapp_entity" "vapp_entity_vappcontrolplane" { target_id = vsphere_vapp_container.vapp_container_controlplane.id container_id = vsphere_vapp_container.vapp_container_cluster.id start_order = 1 start_action = "powerOn" stop_action = "guestShutdown" } resource "vsphere_vapp_entity" "vapp_entity_vappworker" { target_id = vsphere_vapp_container.vapp_container_worker.id container_id = vsphere_vapp_container.vapp_container_cluster.id start_order = 2 start_action = "powerOn" stop_action = "guestShutdown" } ```

hashicorp / terraform-provider-vsphere