Client error when creating nested NDGs using the ise_network_device_group resource

[grg1bbs]

ISE version = 3.2 p4 Provider version = 0.1.9

When trying to create a nested NDG structure with 1 Parent and 2 Child groups using the 'ise_network_device_group' resource, TF throws a Client Error and one of the Child groups fails to create.

If the apply is run a second time, the resource is created successfully, so I suspect this is an issue with a race condition. Opening in issue to determine if this can be addressed within the provider. The child NDG (Child1 vs. Child2) that fails to create is also inconsistent when replicating the issue multiple times.

Example code:

resource "ise_network_device_group" "parent" {
  description = "Parent NDG"
  name        = "Parent#Parent"
  root_group     = "Parent"
}

resource "ise_network_device_group" "child1" {
  depends_on = [
    ise_network_device_group.parent
  ]
  description = "Child1 NDG"
  name        = "Parent#Parent#Child1"
  root_group     = "Parent"
}

resource "ise_network_device_group" "child2" {
  depends_on = [
    ise_network_device_group.parent
  ]
  description = "Child2 NDG"
  name        = "Parent#Parent#Child2"
  root_group     = "Parent"
}

Error output:

❯ tf apply --auto-approve
ise_network_device_group.ndg_deployment_stage: Refreshing state... [id=ba03cea0-b33e-11ee-b966-ead89af9fdb3]
ise_certificate_authentication_profile.certprof_ad: Refreshing state... [id=6fc94630-b33e-11ee-b966-ead89af9fdb3]
ise_allowed_protocols.eaptls: Refreshing state... [id=6fc01e70-b33e-11ee-b966-ead89af9fdb3]
ise_allowed_protocols.mab_eaptls: Refreshing state... [id=6fc0e1c0-b33e-11ee-b966-ead89af9fdb3]
ise_network_device_group.ndg_mm: Refreshing state... [id=d6fd7370-b33f-11ee-b966-ead89af9fdb3]
ise_network_device_group.ndg_lim: Refreshing state... [id=f7fe1700-b33f-11ee-b966-ead89af9fdb3]
ise_identity_source_sequence.iss_ad_cert: Refreshing state... [id=2a9a7060-b33f-11ee-b966-ead89af9fdb3]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # ise_network_device_group.child1 will be created
  + resource "ise_network_device_group" "child1" {
      + description = "Child1 NDG"
      + id          = (known after apply)
      + name        = "Parent#Parent#Child1"
      + root_group  = "Parent"
    }

  # ise_network_device_group.child2 will be created
  + resource "ise_network_device_group" "child2" {
      + description = "Child2 NDG"
      + id          = (known after apply)
      + name        = "Parent#Parent#Child2"
      + root_group  = "Parent"
    }

  # ise_network_device_group.parent will be created
  + resource "ise_network_device_group" "parent" {
      + description = "Parent NDG"
      + id          = (known after apply)
      + name        = "Parent#Parent"
      + root_group  = "Parent"
    }

Plan: 3 to add, 0 to change, 0 to destroy.
ise_network_device_group.parent: Creating...
ise_network_device_group.parent: Creation complete after 1s [id=7b1ae3b0-b341-11ee-b966-ead89af9fdb3]
ise_network_device_group.child2: Creating...
ise_network_device_group.child1: Creating...
ise_network_device_group.child1: Creation complete after 0s [id=7b6ed010-b341-11ee-b966-ead89af9fdb3]
╷
│ Error: Client Error
│ 
│   with ise_network_device_group.child2,
│   on ndg-only.tf line 20, in resource "ise_network_device_group" "child2":
│   20: resource "ise_network_device_group" "child2" {
│ 
│ Failed to configure object (POST), got error: HTTP Request failed: StatusCode 500, Message: Failed to create Network Device Group. Internal
│ error., {
│   "ERSResponse" : {
│     "operation" : "POST-create-networkdevicegroup",
│     "messages" : [ {
│       "title" : "Failed to create Network Device Group. Internal error.",
│       "type" : "ERROR",
│       "code" : "CRUD operation exception"
│     } ],
│     "link" : {
│       "rel" : "related",
│       "href" : "https://ise32-3.ise.trappedunderise.com/ers/config/networkdevicegroup",
│       "type" : "application/xml"
│     }
│   }
│ }

Workaround: If I add a dependency on child1 in the child2 resource block, all resources are created. This is not a true dependency, however, so it should not be needed.

Example workaround code:

resource "ise_network_device_group" "child2" {
  depends_on = [
    ise_network_device_group.parent,
    ise_network_device_group.child1
  ]
  description = "Child2 NDG"
  name        = "Parent#Parent#Child2"
  root_group     = "Parent"
}

[kuba-mazurkiewicz]

Hey @grg1bbs, I experience same issue and i think its a API or backend problem. I face this also for other resources. That when you try to create multiple resources at once or delete multiple resources at once, you might get 500 error status code, without any descriptive information why this happened. There are some workarounds to bypass this. You can either make those fake dependencies like you did, use time_sleep or use --parallelism=1.

Did you try opening SR case to check if that is API issue?

[grg1bbs]

Hi @kuba-mazurkiewicz, I did not open a TAC case as I have little confidence that the dev team would have any intention on investigating/fixing this API issue. The NDG configuration in the GUI is also a bit clunky, so this likely just a race condition.

I will continue using one or more of the workarounds you mentioned. Closing this issue.

[danischm]

Added a workaround to backoff and retry on 400/500 responses (https://github.com/netascode/go-ise/commit/65ff075df70f08b368816fad7194db757e3925c9) which will be included in the next release.

[danischm]

Workaround added in v0.1.10 release.