Failure when removing 'backend' server/service from list

[ajsiersema]

Steps to reproduce :

resources.tf

resource "netscaler_lbmonitor" "lbmonitor" {
  monitorname = "mon_${var.ingress["name"]}"
  type = "${var.backends_config["service_type"]}"
  interval = 350
  resptimeout = 250
}

resource "netscaler_server" "backends" {
  count = "${length(var.backends)}"

  name = "srv_${format("%s_%s", var.ingress["name"], replace(var.backends[count.index], "/[:.]+/", ""))}"
  ipaddress = "${var.backends[count.index]}"
}

resource "netscaler_servicegroup" "backend" {
  servicegroupname = "sg_${var.ingress["name"]}"
  lbvservers = ["${netscaler_lbvserver.lbvserver.name}"]
  servicetype = "${var.backends_config["service_type"]}"
  servicegroupmembers_by_servername = ["${formatlist("%s:%s", netscaler_server.backends.*.name, var.backends_config["port"])}"]
}

resource "netscaler_lbvserver" "lbvserver" {
  name = "lbvs_${var.ingress["name"]}"
  ipv46 = "${var.ingress["ip"]}"
  port = "${var.ingress["port"]}"
  servicetype = "${var.ingress["service_type"]}"
  lbmethod = "ROUNDROBIN"
  persistencetype = "COOKIEINSERT"
}

variables.tf

variable "ns_user" {
  default = "nsroot"
}

variable "ns_password" {
  default = "nsroot"
}

variable "ns_hostname" {
  default = "ns.domain.tld"
}

variable "ingress" {
  type = "map"
  "description" = "name, vip, port and servicetype for the load balancer"
  default {
    port = 443
    service_type = "HTTP"
  }
}

variable "backends_config" {
  type = "map"
  "description" = "port and servicetype for the backend services"
  default {
    backend_port = 80
    service_type = "HTTP"
  }
}

variable "backends" {
  type = "list"
  description = "List of backend services [ipaddress:port]"
}

local.tfvars (first run: create everything)

ingress = {
  name = "ademoapp"
  service_type = "HTTP"
  ip = "10.42.42.100"
  port = 80
}

backends_config = {
  clttimeout = 40
  port = 80
  service_type = "HTTP"
}

backends = [
  "10.42.42.1",
  "10.42.42.2",
  "10.42.42.3",
]

Create the lbvserver and it's backends :

terraform plan -var-file=local.tfvars -out terraform.plan
terraform apply "terraform.plan"

Now change the backends list in local.tfvars to:

backends = [
  "10.42.42.1",
  "10.42.42.3",
]

and run terraform again:

terraform refresh -var-file=local.tfvars
terraform plan -var-file=local.tfvars -out terraform.plan
terraform apply "terraform.plan"

EXPECTED RESULT: 10.42.42.2 removed ACTUAL RESULT:

first try:

terraform plan -var-file=local.tfvars -out terraform.plan
  ~ netscaler_server.backends[1]
      ipaddress:                                    "10.42.42.2" => "10.42.42.3"
      name:                                         "srv_ademoapp_1042422" => "srv_ademoapp_1042423"
  - netscaler_server.backends[2]
  ~ netscaler_servicegroup.backend
      servicegroupmembers_by_servername.#:          "3" => "2"
      servicegroupmembers_by_servername.1839026872: "srv_ademoapp_1042423:80:1" => ""
      servicegroupmembers_by_servername.213590207:  "" => "srv_ademoapp_1042423:80"
      servicegroupmembers_by_servername.2796742708: "" => "srv_ademoapp_1042421:80"
      servicegroupmembers_by_servername.2797699357: "srv_ademoapp_1042422:80:1" => ""
      servicegroupmembers_by_servername.542503859:  "srv_ademoapp_1042421:80:1" => ""
Plan: 0 to add, 2 to change, 1 to destroy.

terraform apply "terraform.plan"
netscaler_server.backends[2]: Destroying... (ID: srv_ademoapp_1042423)
netscaler_server.backends[1]: Modifying... (ID: srv_ademoapp_1042422)
  ipaddress: "10.42.42.2" => "10.42.42.3"
  name:      "srv_ademoapp_1042422" => "srv_ademoapp_1042423"
netscaler_server.backends[2]: Destruction complete after 0s
Error: Error applying plan:
1 error(s) occurred:
* netscaler_server.backends[1]: 1 error(s) occurred:
* netscaler_server.backends.1: Error updating server srv_ademoapp_1042423

idempotent right, so let's try again:

terraform plan -var-file=local.tfvars -out terraform.plan
  ~ netscaler_server.backends[1]
      ipaddress:                                    "10.42.42.2" => "10.42.42.3"
      name:                                         "srv_ademoapp_1042422" => "srv_ademoapp_1042423"
  ~ netscaler_servicegroup.backend
      servicegroupmembers_by_servername.213590207:  "" => "srv_ademoapp_1042423:80"
      servicegroupmembers_by_servername.2796742708: "" => "srv_ademoapp_1042421:80"
      servicegroupmembers_by_servername.2797699357: "srv_ademoapp_1042422:80:1" => ""
      servicegroupmembers_by_servername.542503859:  "srv_ademoapp_1042421:80:1" => ""
Plan: 0 to add, 2 to change, 0 to destroy.

terraform apply "terraform.plan"
netscaler_server.backends[1]: Modifying... (ID: srv_ademoapp_1042422)
  ipaddress: "10.42.42.2" => "10.42.42.3"
  name:      "srv_ademoapp_1042422" => "srv_ademoapp_1042423"
netscaler_server.backends[1]: Modifications complete after 0s (ID: srv_ademoapp_1042422)
Error: Error applying plan:
1 error(s) occurred:
* netscaler_servicegroup.backend: netscaler_servicegroup.backend: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.
Please include the following information in your report:
    Terraform Version: 0.11.11
    Resource ID: netscaler_servicegroup.backend
    Mismatch reason: attribute mismatch: servicegroupmembers_by_servername.213590207
    Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"servicegroupmembers_by_servername.542503859":*terraform.ResourceAttrDiff{Old:"srv_ademoapp_1042421:80:1", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.213590207":*terraform.ResourceAttrDiff{Old:"", New:"srv_ademoapp_1042423:80", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.2796742708":*terraform.ResourceAttrDiff{Old:"", New:"srv_ademoapp_1042421:80", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.2797699357":*terraform.ResourceAttrDiff{Old:"srv_ademoapp_1042422:80:1", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}
    Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"servicegroupmembers_by_servername.542503859":*terraform.ResourceAttrDiff{Old:"srv_ademoapp_1042421:80:1", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.2796742708":*terraform.ResourceAttrDiff{Old:"", New:"srv_ademoapp_1042421:80", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.3020376026":*terraform.ResourceAttrDiff{Old:"", New:"srv_ademoapp_1042422:80", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "servicegroupmembers_by_servername.2797699357":*terraform.ResourceAttrDiff{Old:"srv_ademoapp_1042422:80:1", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}

[giorgos-nikolopoulos]

A little elaboration on the issue and its fix.

The root cause of the error reported was that the provider tried to modify an non modifiable attribute, in this case the attribute "name" of the the "resource_server".

That happened as part of the plan to execute the removal of the middle backend server from the input variables list.

After this failed operation the local state and the state of the target Citrix ADC were mismatched and the second error message was the result.

The fix was to mark this attribute with ForceNew. Now the execution plan will first delete the backend server and then recreate it with the new name.

Also make sure to run terraform apply -parallelism=1.

I noticed that this is required to have correct operation since when re adding the missing back end server resulted in a race condition where the server was created but the servicegroup failed to add it since the nitro API regarded it as not created yet.

The dependency of the netscaler_servicegroup resource on the netscaler_server resource was not enough to resolve this race condition.

[giorgos-nikolopoulos]

@ajsiersema

I made a new release which contains this fix. (link)

Let us know if this works for you.

[ajsiersema]

Works like a charm now. Thanks for the explanation and fix, I've tested with provider v0.11.11 and terraform v0.11.13.

[giorgos-nikolopoulos]

Glad I could help.