re-creation of local peering gateway hangs when it is referenced in a routing rule

biemond commented 5 years ago

Terraform Version

Terraform v0.11.13

OCI Provider Version

3.22

Description:

Our local peering gateway is referenced in a route table. When the opposite peering gateway in a different vcn has changed this LPG will have a not valid established state. In that case terraform wants to re-create the local peering gateway but that is not allowed because it is still being referenced in the route table as a rule and it just hangs.

Workaround: I first need to remove the matching routing rule, after that it works again. There is some circular references between the route table rule and LPG.

So can we do

an update of the LPG (so the OCID does not changes) instead of a re-create
or somehow delete this routing rule first. After this terraform will add the new route table rule again.

thanks Edwin

olkoko commented 5 years ago

We are having exactly the same issue while trying to implement Transit VCN solution. LPGs in the "hub" VCN are associated with Route Table(s) which reference DRG attached to the "hub" VCN. DRG attachment in the "hub" VCN is associated with Route Table which references LPGs in "hub" VCN. This creates circular dependency.

Other than running "terraform destroy" + "terraform apply", the only other workaround we identified so far is to explicitly "taint" resources that need to be re-created on "apply" rather than updated:

terraform taint -module=transit_vcn oci_core_default_route_table.hub
terraform taint -module=transit_vcn oci_core_route_table.hub_to_drg
terraform taint -module=transit_vcn oci_core_drg_attachment.hub_to_drg

The permanent fix would be either going back to immutable Route Tables (to force its recreation when "route_rules" attribute is changed), or introducing new attribute "immutable = true|false" that would instruct OCI provider to always re-create this resource (if this is possible at all).

The argument is, these days when updating association between Route Table and Subnet (or DRG attachment, or LPG) can be updated without forcing recreation of these resources, mutability of Route Table is no longer that important.

alexng-canuck commented 5 years ago

@biemond To help us better understand your use case, can you share the terraform plan output, any Terraform configs, and the debug logs from a terraform apply command?

You can gather the debug logs by prepending your terraform apply with settings like this: OCI_GO_SDK_DEBUG=v TF_LOG=DEBUG terraform apply

Which attribute of the LPG is forcing a re-create in your scenario?

biemond commented 5 years ago

Thanks Alex.

will do on tuesday after easter, I am part of OCI and we hit this with bastion v3

biemond commented 5 years ago

its peer_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa2mota4bkb5aw4fe7yfbe2dbo3mmincz4ljq45nqnkssot3akuajq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa3t3kudless3ttuouz3f4gswrwn4k7nvabwshbp7qxu25mrdm7nia" (forces new resource)



-/+ module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10 (new resource required)
      id:                                       "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaf532vxme3g7p27tpgczmwzsfofgw7htrcbpq2ur3g2cuhiz4rxqa" => <computed> (forces new resource)
      compartment_id:                           "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq" => "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq"
      display_name:                             "bastion_sss_lpg_dev10" => "bastion_sss_lpg_dev10"
2019/04/24 12:25:43 [DEBUG] plugin: waiting for all plugin processes to complete...
      freeform_tags.%:                          "0" => <computed>
2019-04-24T12:25:43.265+0200 [DEBUG] plugin.terraform-provider-oci_v3.23.0_x4: 2019/04/24 12:25:43 [ERR] plugin: plugin server: accept unix /tmp/plugin096959403: use of closed network connection
2019-04-24T12:25:43.268+0200 [DEBUG] plugin: plugin process exited: path=/scratch/projects/sss_terraform/.terraform/plugins/linux_amd64/terraform-provider-oci_v3.23.0_x4
      is_cross_tenancy_peering:                 "false" => <computed>
      peer_advertised_cidr:                     "10.19.0.0/16" => <computed>
      peer_advertised_cidr_details.#:           "0" => <computed>
      peer_id:                                  "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa2mota4bkb5aw4fe7yfbe2dbo3mmincz4ljq45nqnkssot3akuajq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa3t3kudless3ttuouz3f4gswrwn4k7nvabwshbp7qxu25mrdm7nia" (forces new resource)
      peering_status:                           "REVOKED" => <computed>
      peering_status_details:                   "A connection was established in the past, but it has since been destroyed." => <computed>
      route_table_id:                           "" => <computed>
      state:                                    "AVAILABLE" => <computed>
      time_created:                             "2019-04-16 17:11:05.814 +0000 UTC" => <computed>
      vcn_id:                                   "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q" => "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q"

  ~ module.bastion-v3.oci_core_route_table.bastion_jump_route_table
      route_rules.1115362686.destination:       "10.10.0.0/16" => "10.10.0.0/16"
      route_rules.1115362686.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaapqrivdmnouxey47f3ja3emyan6ntieslegytmxs2pk2y6owh23eq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaapqrivdmnouxey47f3ja3emyan6ntieslegytmxs2pk2y6owh23eq"
      route_rules.2514272859.destination:       "all-phx-services-in-oracle-services-network" => "all-phx-services-in-oracle-services-network"
      route_rules.2514272859.destination_type:  "SERVICE_CIDR_BLOCK" => "SERVICE_CIDR_BLOCK"
      route_rules.2514272859.network_entity_id: "ocid1.servicegateway.oc1.phx.aaaaaaaar55nhxowfseslvy7cw5zo5fihvq6yiyuavcyvjor5wcthbwtjgha" => "ocid1.servicegateway.oc1.phx.aaaaaaaar55nhxowfseslvy7cw5zo5fihvq6yiyuavcyvjor5wcthbwtjgha"
      route_rules.2649016511.destination:       "172.16.101.0/26" => "172.16.101.0/26"
      route_rules.2649016511.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaanl4eqmchy7w2zryl53j2hcqg27tremr4aj5a5aggp5yqghzze6zq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaanl4eqmchy7w2zryl53j2hcqg27tremr4aj5a5aggp5yqghzze6zq"
      route_rules.299026434.destination:        "10.11.0.0/16" => "10.11.0.0/16"
      route_rules.299026434.network_entity_id:  "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaogidnbtwsixp5lewkqfc6xjmy73ubw47ckscf5lnfreu6amn6iaq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaogidnbtwsixp5lewkqfc6xjmy73ubw47ckscf5lnfreu6amn6iaq"
      route_rules.3531570314.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaf532vxme3g7p27tpgczmwzsfofgw7htrcbpq2ur3g2cuhiz4rxqa" => ""
      route_rules.3719182765.destination:       "10.12.0.0/16" => "10.12.0.0/16"
      route_rules.3719182765.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaatgr53glhyun5hgedbxhnkudpsbipkx3jjlsrizll3o3rzuneo5pa" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaatgr53glhyun5hgedbxhnkudpsbipkx3jjlsrizll3o3rzuneo5pa"
      route_rules.686046354.destination:        "10.13.0.0/16" => "10.13.0.0/16"
      route_rules.686046354.network_entity_id:  "ocid1.localpeeringgateway.oc1.phx.aaaaaaaajwiziqv7yxd24ivyvxgxpwq65mvbnu23fqn5tf5skhqrh7e54hza" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaajwiziqv7yxd24ivyvxgxpwq65mvbnu23fqn5tf5skhqrh7e54hza"
      route_rules.~122270211.cidr_block:        "" => <computed>
      route_rules.~122270211.destination:       "" => "10.19.0.0/16"
      route_rules.~122270211.destination_type:  "" => <computed>
      route_rules.~122270211.network_entity_id: "" => "${oci_core_local_peering_gateway.bastion_sss_lpg_dev10.id}"

Plan: 1 to add, 1 to change, 1 to destroy.```

biemond commented 5 years ago

terraform_bug.zip

parrneet commented 5 years ago

@biemond Thank you for the details. If possible, please provide the terraform config as well. We are having discussion on planning the fix for this.

parrneet commented 5 years ago

@biemond @olkoko We tried replicating the scenario by updating the peer id for the oci_core_local_peering_gateway and were able to get past this error using create_before_destroy flag in the resource(local peering gateway) config. Can you try adding this flag and see if this works for you.

lifecycle { create_before_destroy = true }

biemond commented 5 years ago

thanks , I will try it out.

biemond commented 5 years ago

It did not work for me , it creates an extra lpg with the same name


Plan: 1 to add, 1 to change, 1 to destroy.

Do you want to perform these actions in workspace "oc1_dev_phoenix_bastion"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10: Creating...
  compartment_id:                 "" => "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq"
  display_name:                   "" => "bastion_sss_lpg_dev10"
  freeform_tags.%:                "" => "<computed>"
  is_cross_tenancy_peering:       "" => "<computed>"
  peer_advertised_cidr:           "" => "<computed>"
  peer_advertised_cidr_details.#: "" => "<computed>"
  peer_id:                        "" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaanjpvnxjsfm4kemn25lf4kcfjdwyezok5gr7w6mzvuugkt7o2xva"
  peering_status:                 "" => "<computed>"
  peering_status_details:         "" => "<computed>"
  route_table_id:                 "" => "<computed>"
  state:                          "" => "<computed>"
  time_created:                   "" => "<computed>"
  vcn_id:                         "" => "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q"

Error: Error applying plan:

1 error(s) occurred:

* module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10: 1 error(s) occurred:

* oci_core_local_peering_gateway.bastion_sss_lpg_dev10: Service error:InvalidParameter. A peering with VCN ocid1.vcn.oc1.phx.aaaaaaaaefm4vw4u2wid3mq5zyaesrujgyog5oztpicaqfc6lprwkifh36fq has already been established.. http status code: 400. Opc request id: 64ae797a034ad9d3af564b9d324faf93/AD3A7D8CAB610643199F44E654FCAEBB/EAE3BD38018D4052BB6410B913DCE13F

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.```

biemond commented 5 years ago

2019-04-26 11_31_13-Oracle Cloud Infrastructure

biemond commented 5 years ago

my tf


data "oci_core_services" "service_gateway_all_oci_services" {
  filter {
    name   = "name"
    values = ["All [A-Za-z0-9]+ Services In Oracle Services Network"]
    regex  = true
  }
}

data "oci_core_vcns" "remote_vcn_dev10" {
  compartment_id = "${var.plane_compartment_id}"
  display_name = "sss_network_${var.env}10"
}

data "oci_core_local_peering_gateways" "bastion_remote_vcn_dev10" {
  compartment_id = "${var.bastion_compartment_id}"
  vcn_id         = "${lookup(data.oci_core_vcns.remote_vcn_dev10.virtual_networks[0], "id")}"
  filter {
    name   = "display_name"
    values = ["sss_bastion_lpg_${var.env}10"]
  }
}

# Creates the Bastion Jump VCN
resource "oci_core_vcn" "bastion_jump_vcn" {
  compartment_id = "${var.bastion_compartment_id}"

  display_name   = "sss_jumpvcn_${var.region_name}"

  cidr_block     = "${var.bastion_jump_vcn_cidr_slash_30}"
  dns_label      = "bj${var.vcn_dns_suffix}" // "bj*" = bastion jump network
}

resource oci_core_service_gateway "bastion_jump_service_gateway" {
  compartment_id = "${var.bastion_compartment_id}"
  vcn_id         = "${oci_core_vcn.bastion_jump_vcn.id}"

  display_name   = "bastion_jump_service_gateway"

  services {
    service_id   = "${lookup(data.oci_core_services.service_gateway_all_oci_services.services[0], "id")}"
  }
}

resource "oci_core_route_table" "bastion_jump_route_table" {
  compartment_id = "${var.bastion_compartment_id}"
  vcn_id         = "${oci_core_vcn.bastion_jump_vcn.id}"

  display_name   = "bastion_jump_route_table"

  route_rules    = [
    {
      destination_type  = "SERVICE_CIDR_BLOCK"
      destination       = "${lookup(data.oci_core_services.service_gateway_all_oci_services.services[0], "cidr_block")}"
      network_entity_id = "${oci_core_service_gateway.bastion_jump_service_gateway.id}"
    },
    {
      destination       = "${oci_core_local_peering_gateway.bastion_jump_lpg.peer_advertised_cidr}"
      #destination       = "${local.temporary_unpeered_lpg_default_cidr}"
      network_entity_id = "${oci_core_local_peering_gateway.bastion_jump_lpg.id}"
    },
    {
      destination       = "10.19.0.0/16"
      network_entity_id = "${oci_core_local_peering_gateway.bastion_sss_lpg_dev10.id}"
    },            
  ]
}

resource "oci_core_local_peering_gateway" "bastion_sss_lpg_dev10" {
    count          = "${var.env == "dev" ? 1 : 0}"
    compartment_id = "${var.bastion_compartment_id}"
    vcn_id         = "${oci_core_vcn.bastion_jump_vcn.id}"

    display_name   = "bastion_sss_lpg_dev10"

    peer_id        = "${lookup(data.oci_core_local_peering_gateways.bastion_remote_vcn_dev10.local_peering_gateways[0], "id")}"
    lifecycle { create_before_destroy = true }     
}

parrneet commented 5 years ago

@biemond Please confirm that create_before_destroy is already applied to the config of LPG before the opposite LPG changes. If create_before_destroy is ture for the LPG, Terraform operations should follow below sequence:

Create the new LPG with updated peer_id
Update the reference in route table to the new LPG
Delete the old LPG. We verified this behavior by updating the 'peer_id' of LPG to a third LPG (different vcn than the 2 existing).

biemond commented 5 years ago

Ok thanks let me retry that on monday. Will let you know.

biemond commented 5 years ago

It did not work, because the old not valid lpg is somehow still connected to this vcn and it is not allowed to create the 2nd lpg to the same vcn. So the workaround is to delete the old lpg after deleting the routing rule . so this workaround made it worse. In the old workaround I only needed to remove the rules in the route table. now I also need to delete the old lpg.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10 (deposed) (new resource required)
      id:                                       "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa32pmzhmhjsijbnk3iwwqhffaza7okn3rmch2qegspkwrga43ckoq" => <computed> (forces new resource)
      compartment_id:                           "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq" => "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq"
      display_name:                             "bastion_sss_lpg_dev10" => "bastion_sss_lpg_dev10"
      freeform_tags.%:                          "0" => <computed>
      is_cross_tenancy_peering:                 "false" => <computed>
      peer_advertised_cidr:                     "10.19.0.0/16" => <computed>
      peer_advertised_cidr_details.#:           "0" => <computed>
      peer_id:                                  "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaxw6iqjvgi6ptn22ybozqdflft7i7yesgscmm3yjithz3dmwomwsq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa5hsz5kjwr3ceg2hfyrhalbnlfcnjeiylmx3khbhbuusj6wdxh6ra" (forces new resource)
      peering_status:                           "REVOKED" => <computed>
      peering_status_details:                   "A connection was established in the past, but it has since been destroyed." => <computed>
      route_table_id:                           "" => <computed>
      state:                                    "AVAILABLE" => <computed>
      time_created:                             "2019-04-29 14:43:45.876 +0000 UTC" => <computed>
      vcn_id:                                   "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q" => "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q"

  ~ module.bastion-v3.oci_core_route_table.bastion_jump_route_table
      route_rules.#:                            "6" => "7"
      route_rules.1631680466.destination:       "10.13.0.0/16" => "10.13.0.0/16"
      route_rules.1631680466.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaw5cbyxhqoswucukovruc4rbu5z3exeyfnt6oy6a7275yheg43vwq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaw5cbyxhqoswucukovruc4rbu5z3exeyfnt6oy6a7275yheg43vwq"
      route_rules.2514272859.destination:       "all-phx-services-in-oracle-services-network" => "all-phx-services-in-oracle-services-network"
      route_rules.2514272859.destination_type:  "SERVICE_CIDR_BLOCK" => "SERVICE_CIDR_BLOCK"
      route_rules.2514272859.network_entity_id: "ocid1.servicegateway.oc1.phx.aaaaaaaar55nhxowfseslvy7cw5zo5fihvq6yiyuavcyvjor5wcthbwtjgha" => "ocid1.servicegateway.oc1.phx.aaaaaaaar55nhxowfseslvy7cw5zo5fihvq6yiyuavcyvjor5wcthbwtjgha"
      route_rules.2649016511.destination:       "172.16.101.0/26" => "172.16.101.0/26"
      route_rules.2649016511.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaanl4eqmchy7w2zryl53j2hcqg27tremr4aj5a5aggp5yqghzze6zq" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaanl4eqmchy7w2zryl53j2hcqg27tremr4aj5a5aggp5yqghzze6zq"
      route_rules.2696533175.destination:       "10.10.0.0/16" => "10.10.0.0/16"
      route_rules.2696533175.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaidnh5nkeduqd3mjmerzpy6e5w7aiz7tn6rxeyqd2umy2fxc2u2ga" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaidnh5nkeduqd3mjmerzpy6e5w7aiz7tn6rxeyqd2umy2fxc2u2ga"
      route_rules.3403350637.destination:       "10.12.0.0/16" => "10.12.0.0/16"
      route_rules.3403350637.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaquerakx3bwaboi3sk6hzhftcuyrfn2fwgjoevyiniqznusnethva" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaaquerakx3bwaboi3sk6hzhftcuyrfn2fwgjoevyiniqznusnethva"
      route_rules.3450915270.destination:       "10.11.0.0/16" => "10.11.0.0/16"
      route_rules.3450915270.network_entity_id: "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa6ygznw3n43sw664bo657i3hu5ya4wm45zl4jmdt73obmiivixrpa" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa6ygznw3n43sw664bo657i3hu5ya4wm45zl4jmdt73obmiivixrpa"
      route_rules.~122270211.cidr_block:        "" => <computed>
      route_rules.~122270211.destination:       "" => "10.19.0.0/16"
      route_rules.~122270211.destination_type:  "" => <computed>
      route_rules.~122270211.network_entity_id: "" => "${oci_core_local_peering_gateway.bastion_sss_lpg_dev10.id}"

Plan: 1 to add, 1 to change, 1 to destroy.

Do you want to perform these actions in workspace "oc1_dev_phoenix_bastion"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10: Creating...
  compartment_id:                 "" => "ocid1.compartment.oc1..aaaaaaaavwizrrgtbjsdo4mduuutusr3uhimm6kqylmll7lt4jwl6zg3bslq"
  display_name:                   "" => "bastion_sss_lpg_dev10"
  freeform_tags.%:                "" => "<computed>"
  is_cross_tenancy_peering:       "" => "<computed>"
  peer_advertised_cidr:           "" => "<computed>"
  peer_advertised_cidr_details.#: "" => "<computed>"
  peer_id:                        "" => "ocid1.localpeeringgateway.oc1.phx.aaaaaaaa5hsz5kjwr3ceg2hfyrhalbnlfcnjeiylmx3khbhbuusj6wdxh6ra"
  peering_status:                 "" => "<computed>"
  peering_status_details:         "" => "<computed>"
  route_table_id:                 "" => "<computed>"
  state:                          "" => "<computed>"
  time_created:                   "" => "<computed>"
  vcn_id:                         "" => "ocid1.vcn.oc1.phx.aaaaaaaa67knx4a5k5fvfzfixft4qg6q7jlbc6gykhqbl2mjwkerxcjtl32q"

Error: Error applying plan:

1 error(s) occurred:

* module.bastion-v3.oci_core_local_peering_gateway.bastion_sss_lpg_dev10: 1 error(s) occurred:

* oci_core_local_peering_gateway.bastion_sss_lpg_dev10: Service error:InvalidParameter. A peering with VCN ocid1.vcn.oc1.phx.aaaaaaaaefm4vw4u2wid3mq5zyaesrujgyog5oztpicaqfc6lprwkifh36fq has already been established.. http status code: 400. Opc request id: dee317871af9df474dbeb9770636958c/301259BA36B8C8BEB66725D31E4FB5EA/178E0E0AD4934C21A70FB2583181AFF8

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

other side is ready to peer ( was deleted and re-created again by terraform) 2019-04-29 16_58_26-Oracle Cloud Infrastructure

the bastion side 2019-04-29 17_00_32-Oracle Cloud Infrastructure

rcohenma commented 5 years ago

Hello biemond,

I've looked into it. terraform does not hang in the specified scenario. Our provider is retrying on a 409 error from the service which prevents the LPG from being destroyed when it is associated with a route_table.

The create_before_destroy solution works only when the peering_status of the LPG is not PEERED. This solution makes it so that a new LPG is created and the route_table is updated with the new ID before the old LPG is destroyed (the operation succeeds because the old LPG is no longer associated with the RouteTable at this point).

However in the case of an LPG where the peering_status is PEERED, If the create_before_destroy is set to true, the connect operation after the creation of the new LPG will fail because the service will complain that there is already an LPG connected to that VCN.

The create_before_destroy solution on the requestor LPG (the one with the peer_id) can work for you if the acceptor LPG is destroyed first so that the peering status of the requestor LPG becomes REVOKED before the creation of the new LPG.

Unfortunately you cannot currently use conditionals based on the peering status in the create_before_destroy property: https://github.com/hashicorp/terraform/issues/3116

We don't see a solution that we can implement in the provider that addresses both cases. We have contacted the LPG service team to see if they can change the service so that deleting and recreating the LPG can be avoided when updating the peer_id.

Based on how terraform currently works there is no safe way to programmatically update the route_rule when deleting and recreating the LPG.

We will wait for a response from the service team.

vikingsloth commented 4 years ago

This is still an issue. Any updates from the service team?

ravinitp commented 1 year ago

We are very sorry that we couldn't respond to each and every issue reported on GitHub. Although we have refined the process to prioritize customer issues on GitHub, since this issue was reported a while ago, there is a good chance it may have been fixed in the latest version of Terraform Provider OCI. If you are still experiencing this issue, please create a new issue and label it as Bug.

oracle / terraform-provider-oci