search

vmware / terraform-provider-vcd

Terraform VMware Cloud Director provider
https://www.terraform.io/docs/providers/vcd/
Mozilla Public License 2.0
149 stars 111 forks source link

vcd_edgegateway: changing suballocate_pool requires the edge to be replaced #647

Open carmine73 opened 3 years ago

carmine73 commented 3 years ago

Terraform Version

Terraform v0.14.7
+ provider registry.terraform.io/vmware/vcd v3.1.0

Affected Resource(s)

Terraform Configuration Files

## edge gateway 1
resource "vcd_edgegateway" "edge1" {
  org = vcd_org.org1.name
  vdc = vcd_org_vdc.vdc1.name

  name          = local.vcd_edge_1.name
  description   = local.vcd_edge_1.descr
  configuration = local.vcd_edge_1.size

  # primary external network
  external_network {
    name = data.vcd_external_network.ext_net_internet.name
    subnet {
      use_for_default_route = true
      # comment ip_address for auto-assigned ip or set pub_ext_ip = ""
      ip_address            = local.vcd_edge_1.ext_ip
      gateway               = data.vcd_external_network.ext_net_internet.ip_scope[0].gateway
      netmask               = data.vcd_external_network.ext_net_internet.ip_scope[0].netmask

      # dynamic suballocation
      dynamic "suballocate_pool" {
        for_each = local.vcd_edge_1.ext_pools

        content {
          start_address = suballocate_pool.value[0]
          end_address   = suballocate_pool.value[1]
        }
      }
    }
  }
}

Steps to Reproduce

  1. terraform plan
  2. terraform apply
  3. Change the value of the configuration file to have: end_address = "10.10.10.13" -> "10.10.10.14"
  4. terraform plan

Debug Output

The edge creation is ok, but when I try to update the configuration I got:

Terraform will perform the following actions:

  # vcd_edgegateway.edge1 must be replaced
-/+ resource "vcd_edgegateway" "edge1" {
      ~ default_external_network_ip     = "10.10.10.12" -> (known after apply)
      ~ external_network_ips            = [
          - "10.10.10.12",
        ] -> (known after apply)
      ~ id                              = "urn:vcloud:gateway:12da6a99-d424-4435-90cb-450deaf01602" -> (known after apply)
        name                            = "org-template_2_org_ext_nat_1"
      ~ use_default_route_for_dns_relay = false -> (known after apply)
        # (14 unchanged attributes hidden)

      - external_network { # forces replacement
          - enable_rate_limit   = false -> null
          - incoming_rate_limit = 0 -> null
          - name                = "Internet_01-RZ" -> null
          - outgoing_rate_limit = 0 -> null

          - subnet {
              - gateway               = "10.10.10.1" -> null
              - netmask               = "255.255.255.128" -> null
              - use_for_default_route = true -> null

              - suballocate_pool {
                  - end_address   = "10.10.10.13" -> null
                  - start_address = "10.10.10.13" -> null
                }
            }
        }
      + external_network { # forces replacement
          + enable_rate_limit   = false
          + incoming_rate_limit = 0
          + name                = "Internet_01-RZ"
          + outgoing_rate_limit = 0

          + subnet {
              + gateway               = "10.10.10.1"
              + netmask               = "255.255.255.128"
              + use_for_default_route = true

              + suballocate_pool {
                  + end_address   = "10.10.10.14"
                  + start_address = "10.10.10.13"
                }
            }
        }
    }

Expected Behavior

Suballocated pool is changed in place

Actual Behavior

Edge is going to be replaced. Since there are routed networks attached, I got an error:

vcd_edgegateway.edge1: Destroying... [id=urn:vcloud:gateway:12da6a99-d424-4435-90cb-450deaf01602]

Error: error deleting edge gateway: API Error: 400: [ 13fec1cb-61c5-49ca-9ed4-17363caf14fb ] Edge gateway org-template_2_org_ext_nat_1 has Org VDC networks attached to it. Please delete the networks before attempting to delete the edge gateway.

References

@didainius, we talked about this at the end of 2019 https://github.com/vmware/terraform-provider-vcd/issues/323#issuecomment-562034891

carmine73 commented 3 years ago

Importing an edge gateway does not pupulate the external_network list in the state file, so every subsequent action on that edge forces the replacement, even if the configuration of the external network is unchanged. Here you can see the state file vdc_edgegateway block (after import and refresh) with "external_network": []

    {
      "mode": "managed",
      "type": "vcd_edgegateway",
      "name": "edge1",
      "provider": "provider[\"registry.terraform.io/vmware/vcd\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "configuration": "compact",
            "default_external_network_ip": "10.10.10.11",
            "description": "edge gw created by terraform",
            "distributed_routing": false,
            "external_network": [],
            "external_network_ips": [
              "10.10.10.11"
            ],
            "fips_mode_enabled": false,
            "fw_default_rule_action": "deny",
            "fw_default_rule_logging_enabled": false,
            "fw_enabled": true,
            "ha_enabled": false,
            "id": "urn:vcloud:gateway:1cd610f1-24f0-4f41-a3a8-dcffg00a1458",
            "lb_acceleration_enabled": false,
            "lb_enabled": false,
            "lb_logging_enabled": false,
            "lb_loglevel": "info",
            "name": "org-template_1_edge_1",
            "org": "org-template_1",
            "use_default_route_for_dns_relay": false,
            "vdc": "org-template_1_ALL_1"
          },
          "sensitive_attributes": [],
          "private": "eyJzY2hlbWFfaaVzb2lvbiI6IjAifQ==",
          "dependencies": [
            "data.vcd_edgegateway.my-edge1",
            "data.vcd_org.my-org1",
            "data.vcd_org_vdc.my-vdc1",
            "vcd_org.org1",
            "vcd_org_vdc.vdc1"
          ]
        }
      ]
    },
carmine73 commented 3 years ago

Just a note: extending the edge suballocate_pool is a quite frequent operation done by vcloud sys admins and should be done in place. I'm sorry nobody add comments on this issue after 1 month :-(

dataclouder commented 3 years ago

Sorry for the extended silence. We are dealing with different priorities at the moment, and this issue has not made the cut yet.

carmine73 commented 3 years ago

I've realized that every change on a NSX-v edge gateway requires the edge to be replaced, even the description

Terraform will perform the following actions:

  # vcd_edgegateway.edge1 must be replaced
-/+ resource "vcd_edgegateway" "edge1" {
      ~ default_external_network_ip     = "10.23.96.115" -> (known after apply)
      ~ description                     = "edge gw created by terraform" -> "edge gw created/modified by terraform" # forces replacement
      ~ external_network_ips            = [
          - "10.23.96.115",
        ] -> (known after apply)
      ~ id                              = "urn:vcloud:gateway:158b72e9-9b35-47d2-ad02-63c45956d0cdc" -> (known after apply)
        name                            = "org-testcoll_3_edge_1"
      ~ use_default_route_for_dns_relay = false -> (known after apply)
        # (13 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

while almost every change on edge gateway can be done with:

PUT https://<VCD>/network/edges/{id}