race condition for destroy of ibm_is_virtual_endpoint_gateway and ibm_is_security_group

powellquiring commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

bug-vpc-endpoint-gateway-security-group $ tf version
Terraform v1.1.9
on darwin_amd64
+ provider registry.terraform.io/hashicorp/time v0.7.2
+ provider registry.terraform.io/ibm-cloud/ibm v1.41.0

Affected Resource(s)

ibm_is_security_group
ibm_is_virtual_endpoint_gateway

Terraform Configuration Files

Plan: 0 to add, 0 to change, 4 to destroy.
ibm_is_security_group_rule.cloud_egress_cos: Destroying... [id=r006-4886bf3e-ff62-42a5-a7dc-c21837e861c1.r006-8a4a8d7d-8ce7-4c6c-9eea-50b4ecdd3269]
ibm_is_security_group_rule.cloud_ingress_cos: Destroying... [id=r006-4886bf3e-ff62-42a5-a7dc-c21837e861c1.r006-5c313df6-e708-4bf5-ac75-91caedf11111]
ibm_is_virtual_endpoint_gateway.cos: Destroying... [id=r006-492dbaf1-5f86-4519-8ad3-1628434fbc6e]
ibm_is_virtual_endpoint_gateway.cos: Destruction complete after 0s
ibm_is_security_group_rule.cloud_egress_cos: Destruction complete after 1s
ibm_is_security_group_rule.cloud_ingress_cos: Destruction complete after 1s
ibm_is_security_group.cos: Destroying... [id=r006-4886bf3e-ff62-42a5-a7dc-c21837e861c1]
╷
│ Error: [ERROR] Error Deleting Security Group Targets : The specified endpoint gateway is not attached to any other security groups.
│ {
│     "StatusCode": 409,
│     "Headers": {
│         "Cache-Control": [
│             "max-age=0, no-cache, no-store, must-revalidate"
│         ],
│         "Cf-Cache-Status": [
│             "DYNAMIC"
│         ],
│         "Cf-Ray": [
│             "70ae38610c4a13e0-SEA"
│         ],
│         "Content-Length": [
│             "268"
│         ],
│         "Content-Type": [
│             "application/json"
│         ],
│         "Date": [
│             "Fri, 13 May 2022 20:42:22 GMT"
│         ],
│         "Expect-Ct": [
│             "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""
│         ],
│         "Expires": [
│             "-1"
│         ],
│         "Pragma": [
│             "no-cache"
│         ],
│         "Server": [
│             "cloudflare"
│         ],
│         "Strict-Transport-Security": [
│             "max-age=31536000; includeSubDomains"
│         ],
│         "Vary": [
│             "Accept-Encoding"
│         ],
│         "X-Content-Type-Options": [
│             "nosniff"
│         ],
│         "X-Request-Id": [
│             "1e83b7e7-f706-462a-9d1d-b15b512cd8c7"
│         ],
│         "X-Xss-Protection": [
│             "1; mode=block"
│         ]
│     },
│     "Result": {
│         "errors": [
│             {
│                 "code": "conflict_field",
│                 "message": "The specified endpoint gateway is not attached to any other security groups.",
│                 "target": {
│                     "name": "id",
│                     "type": "parameter",
│                     "value": "r006-492dbaf1-5f86-4519-8ad3-1628434fbc6e"
│                 }
│             }
│         ],
│         "trace": "1e83b7e7-f706-462a-9d1d-b15b512cd8c7"
│     },
│     "RawResult": null
│ }
│
│

Race condition between the destroy of ibm_is_virtual_endpoint_gateway which is reported as destroyed, and later the destroy of the ibm_is_security_group_rule which fails because it is the only SG attached to the endpoint_gateway.

cat sgeg.tf.bu

resource "ibm_is_security_group" "cos" {
  name           = "${local.BASENAME_CLOUD}-cos"
  vpc            = ibm_is_vpc.cloud.id
  resource_group = data.ibm_resource_group.all_rg.id
}

resource "ibm_is_security_group_rule" "cloud_ingress_cos" {
  group     = ibm_is_security_group.cos.id
  direction = "inbound"
  remote    = "10.0.0.0/8" // on prem and cloud
  tcp {
    port_min = 443
    port_max = 443
  }
}
resource "ibm_is_security_group_rule" "cloud_egress_cos" {
  group     = ibm_is_security_group.cos.id
  direction = "outbound"
  remote    = "10.0.0.0/8" // on prem and cloud
}

resource "ibm_is_virtual_endpoint_gateway" "cos" {
  vpc            = ibm_is_vpc.cloud.id
  name           = "${local.BASENAME_CLOUD}-cos"
  resource_group = data.ibm_resource_group.all_rg.id
  target {
    crn           = "crn:v1:bluemix:public:cloud-object-storage:global:::endpoint:${local.cos_endpoint}"
    resource_type = "provider_cloud_service"
  }

  security_groups = [ibm_is_security_group.cos.id]

  # one Reserved IP per zone in the VPC
  ips {
    subnet = ibm_is_subnet.cloud.id
    name   = "cos"
  }
  tags = local.tags
}

Debug Output

Steps to Reproduce

cd /tmp
git clone https://github.com/powellquiring/tfbugs
cd tfbugs/bug-vpc-endpoint-gateway-security-group
terraform init
terraform apply
mv sgeg.tf sgeg.tf.bu
terraform apply

SunithaGudisagarIBM commented 2 years ago

Hi @powellquiring,

Thanks for reporting the issue.

As a first step of my analysis i tried to reproduce the issue in test.cloud.ibm account and creation and destroy worked fine.

Also,

I ran the same test case in prod env i.e cloud.ibm account and still everything worked fine for me.

Please find the attached template and the result of the testing.

Can you please try again with the latest provider and if the issue still persist we can have a call to discus on the issue.

resource "ibm_is_security_group" "cos" {
  name = "security-group-testing"
  #   vpc  = "r134-7b4a313e-d8cc-4d50-92ea-7dd315dadd9b"
  vpc = "r006-b0031fff-c6bb-4a4d-9afd-6c3fc9b7fb5b"
  #   resource_group = data.ibm_resource_group.all_rg.id
}

resource "ibm_is_security_group_rule" "cloud_ingress_cos" {
  group     = ibm_is_security_group.cos.id
  direction = "inbound"
  remote    = "10.0.0.0/8" // on prem and cloud
  tcp {
    port_min = 443
    port_max = 443
  }
}
resource "ibm_is_security_group_rule" "cloud_egress_cos" {
  group     = ibm_is_security_group.cos.id
  direction = "outbound"
  remote    = "10.0.0.0/8" // on prem and cloud
}

resource "ibm_is_virtual_endpoint_gateway" "endpoint_gateway" {
  name = "virtual-endpoint-gateway-cos"
  target {
    name          = "ibm-dns-server2"
    resource_type = "provider_infrastructure_service"
  }
  #   vpc             = "r134-7b4a313e-d8cc-4d50-92ea-7dd315dadd9b"
  vpc             = "r006-b0031fff-c6bb-4a4d-9afd-6c3fc9b7fb5b"
  security_groups = [ibm_is_security_group.cos.id]

}

powellquiring commented 2 years ago

I was able to reproduce using the steps provided, they are repeated below. Did you try these steps?

cd /tmp
git clone https://github.com/powellquiring/tfbugs
cd tfbugs/bug-vpc-endpoint-gateway-security-group
terraform init
terraform apply
mv sgeg.tf sgeg.tf.bu
terraform apply

`

SunithaGudisagarIBM commented 2 years ago

@powellquiring I followed the steps you have mentioned and i still dont face any issue, can you please help me to reproduce the issue.

Please find the screen shot of the result:

IBM-Cloud / terraform-provider-ibm