[Bug]: Deletion of `aws_lb` doesn't wait for disassociating EIPs from network interface

Terraform Core Version

1.5.6

AWS Provider Version

5.17.0

Affected Resource(s)

aws_lb
aws_eip

Expected Behavior

Deletion of the LB should wait until the underlying network interface as well as the associated EIPs to that interface are deleted / disassociated before reporting deletion success back to Terraform.

Actual Behavior

The assumption here is that deletion of the LB reports as successful even if the underlying network interface and it's EIP associations have not yet been deployed. Terraform goes on with the deletion of resources until the EIP, where it errors, as the EIP has not yet been disassociated.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_eip" "lb" {
  for_each = toset([var.zone])
  domain   = "vpc"
  tags     = merge(local.tags, { "constellation-ip-endpoint" = each.key == var.zone ? "legacy-primary-zone" : "additional-zone" })
}

resource "aws_lb" "front_end" {
  name               = "${local.name}-loadbalancer"
  internal           = false
  load_balancer_type = "network"
  tags               = local.tags
  security_groups    = [aws_security_group.security_group.id]

  dynamic "subnet_mapping" {
    # for_each = toset(module.public_private_subnet.all_zones)
    for_each = toset([var.zone])
    content {
      subnet_id     = module.public_private_subnet.public_subnet_id[subnet_mapping.key]
      allocation_id = aws_eip.lb[subnet_mapping.key].id
    }
  }
  enable_cross_zone_load_balancing = true

  lifecycle {
    ignore_changes = [security_groups]
  }
}

Steps to Reproduce

Apply the configuration
Delete the configuration
Repeat until deletion fails

Debug Output

LB Deletion:

2023-09-21T06:40:59.578Z [DEBUG] aws_lb.front_end: applying the planned Delete change
2023-09-21T06:40:59.579Z [INFO]  provider.terraform-provider-aws_v5.6.2_x5: [INFO] Deleting LB: arn:aws:elasticloadbalancing:us-east-2:795746500882:loadbalancer/net/e2e-test-78133482-loadbalancer/21769f4e10986d92
2023-09-21T06:40:59.580Z [DEBUG] provider.terraform-provider-aws_v5.6.2_x5: HTTP Request Sent: http.url=https://elasticloadbalancing.us-east-2.amazonaws.com/ http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.7 (+https://www.terraform.io) terraform-provider-aws/5.6.2 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.293 (go1.20.5; linux; amd64) HashiCorp-terraform-exec/0.18.1" aws.sdk=aws-sdk-go http.flavor=1.1 http.request.body="Acti*****************ncer&Load***********=arn%3Aaws%3Ael**************cing%3Aus-east-2%3A795746500882%3Aloadbalancer%2Fnet%2Fe2e-test-78133482-loadbalancer%2F21**********6d92&Version=2015-12-01
" tf_resource_type=aws_lb @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.31/logger.go:96 aws.operation=DeleteLoadBalancer http.method=POST http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************L2JM********1/us-east-2/el*******************aws4_request, Sign*************tent-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.request.header.x_amz_date=20230921T064059Z http.request.header.x_amz_security_token=***** tf_mux_provider=*schema.GRPCProviderServer aws.region=us-east-2 aws.service="Elastic Load Balancing v2" tf_req_id=27425603-8bb0-b574-08b8-6bd68ceb10b3 net.peer.name=elasticloadbalancing.us-east-2.amazonaws.com tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ApplyResourceChange @module=aws http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" http.request_content_length=193 timestamp=2023-09-21T06:40:59.580Z
2023-09-21T06:40:59.711Z [DEBUG] provider.terraform-provider-aws_v5.6.2_x5: HTTP Response Received: aws.operation=DeleteLoadBalancer aws.service="Elastic Load Balancing v2" http.response.body="<Dele******************onse xmlns="http://el**************cing.amazonaws.com/doc/2015-12-01/">
  <Dele*****************ult/>
  <Resp********data>
    <RequestId>c578865e-d3f4-48e0-9753-5395fb45bda1</RequestId>
  </Res*********data>
</Del*******************onse>
" http.response.header.content_type=text/xml @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.31/logger.go:144 aws.sdk=aws-sdk-go tf_req_id=27425603-8bb0-b574-08b8-6bd68ceb10b3 tf_resource_type=aws_lb tf_rpc=ApplyResourceChange tf_mux_provider=*schema.GRPCProviderServer tf_provider_addr=registry.terraform.io/hashicorp/aws @module=aws aws.region=us-east-2 http.duration=130 http.response.header.date="Thu, 21 Sep 2023 06:40:59 GMT" http.response.header.x_amzn_requestid=c578865e-d3f4-48e0-9753-5395fb45bda1 http.response_content_length=262 http.status_code=200 timestamp=2023-09-21T06:40:59.711Z

EIP Deletion:

2023-09-21T06:41:37.606Z [INFO]  Starting apply for aws_eip.lb["us-east-2c"]
2023-09-21T06:41:37.606Z [DEBUG] aws_eip.lb["us-east-2c"]: applying the planned Delete change
2023-09-21T06:41:37.607Z [DEBUG] provider.terraform-provider-aws_v5.6.2_x5: HTTP Request Sent: http.request.header.content_type="application/x-www-form-urlencoded; charset=utf-8" tf_mux_provider=*schema.GRPCProviderServer tf_req_id=d7122774-0f10-5054-f3c0-b9cd722be928 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.31/logger.go:96 aws.service=EC2 http.request.body="Acti******************ress&Asso**************ssoc-0ca2*********37fe&Version=2016-11-15
" http.method=POST http.request.header.authorization="AWS4-HMAC-SHA256 Credential=ASIA************L2JM********1/us-east-2/ec2/aws4_request, Sign*************tent-length;content-type;host;x-amz-date;x-amz-security-token, Signature=*****" http.url=https://ec2.us-east-2.amazonaws.com/ tf_provider_addr=registry.terraform.io/hashicorp/aws aws.region=us-east-2 http.request.header.x_amz_date=20230921T064137Z http.request.header.x_amz_security_token=***** http.request_content_length=86 @module=aws http.flavor=1.1 net.peer.name=ec2.us-east-2.amazonaws.com tf_resource_type=aws_eip aws.operation=DisassociateAddress aws.sdk=aws-sdk-go http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.5.7 (+https://www.terraform.io) terraform-provider-aws/5.6.2 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.293 (go1.20.5; linux; amd64) HashiCorp-terraform-exec/0.18.1" timestamp=2023-09-21T06:41:37.607Z
2023-09-21T06:41:38.534Z [DEBUG] provider.terraform-provider-aws_v5.6.2_x5: HTTP Response Received: tf_req_id=d7122774-0f10-5054-f3c0-b9cd722be928 tf_rpc=ApplyResourceChange @module=aws aws.operation=DisassociateAddress aws.service=EC2 http.response.body="<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>AuthFailure</Code><Message>You do not have permission to access the specified resource.</Message></Error></Errors><RequestID>325f269b-0dc7-4fc7-b329-fbddb5425cb7</RequestID></Response>
" http.response.header.cache_control="no-cache, no-store" http.response.header.x_amzn_requestid=325f269b-0dc7-4fc7-b329-fbddb5425cb7 tf_mux_provider=*schema.GRPCProviderServer tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_eip aws.sdk=aws-sdk-go http.duration=926 http.response.header.date="Thu, 21 Sep 2023 06:41:37 GMT" http.response.header.server=AmazonEC2 http.response.header.strict_transport_security="max-age=31536000; includeSubDomains" @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.31/logger.go:144 aws.region=us-east-2 http.response.header.content_type=text/xml;charset=UTF-8 http.response.header.vary=accept-encoding http.status_code=400 timestamp=2023-09-21T06:41:38.534Z
2023-09-21T06:41:38.534Z [ERROR] provider.terraform-provider-aws_v5.6.2_x5: Response contains error diagnostic: diagnostic_detail= diagnostic_severity=ERROR diagnostic_summary="disassociating EC2 EIP (eipassoc-0ca2c846c154a37fe): AuthFailure: You do not have permission to access the specified resource.
    status code: 400, request id: 325f269b-0dc7-4fc7-b329-fbddb5425cb7" tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d7122774-0f10-5054-f3c0-b9cd722be928 @caller=github.com/hashicorp/terraform-plugin-go@v0.17.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto tf_resource_type=aws_eip tf_rpc=ApplyResourceChange timestamp=2023-09-21T06:41:38.534Z
2023-09-21T06:41:38.539Z [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2023-09-21T06:41:38.539Z [ERROR] vertex "aws_eip.lb[\"us-east-2c\"] (destroy)" error: disassociating EC2 EIP (eipassoc-0ca2c846c154a37fe): AuthFailure: You do not have permission to access the specified resource.

Panic Output

No response

Important Factoids

We ran into the deletion failures in about 3% of cases.

References

No response

hashicorp / terraform-provider-aws