Subnet can't be deleted if used in a VPC endpoint

[dan-hook]

Description

Deploying a VPC with an endpoint with 3 AZs, then reducing the number to 2 AZs fails.

• [x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

• Terraform version: 1.6.1

• Provider version(s): registry.terraform.io/hashicorp/aws v5.22.0

Reproduction Code [Required]

provider "aws" {
  region  = local.region
  profile = local.profile
  default_tags {
    tags = local.tags
  }
}

data "aws_availability_zones" "available" {}

locals {
  name    = "ex-${basename(path.cwd)}"
  region  = "us-east-2"
  profile = "sandbox-daniel.hook.AWSAdministratorAccess"

  vpc_cidr = "10.0.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Example    = local.name
    GithubRepo = "terraform-aws-vpc"
    GithubOrg  = "terraform-aws-modules"
  }
}

module "vpc" {
  source = "../../"

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k)]
}

module "vpc_endpoints" {
  source = "../../modules/vpc-endpoints"

  vpc_id = module.vpc.vpc_id

  create_security_group      = true
  security_group_name_prefix = "${local.name}-vpc-endpoints-"
  security_group_description = "VPC endpoint security group"
  security_group_rules = {
    ingress_https = {
      description = "HTTPS from VPC"
      cidr_blocks = [module.vpc.vpc_cidr_block]
    }
  }

  endpoints = {
    ec2messages = {
      service             = "ec2messages"
      private_dns_enabled = true
      subnet_ids          = module.vpc.private_subnets
    }
  }
}

Steps to reproduce the behavior:

Deploy the module as above. Change the line: azs = slice(data.aws_availability_zones.available.names, 0, 3) to azs = slice(data.aws_availability_zones.available.names, 0, 2)

Plan the module. It will show that it's deleting and changing a number of resources. Apply the plan.

Expected behavior

The plan is applied, deleting the third subnet.

Actual behavior

It will say "Still destroying" for about 20 minutes, before it gives up saying the subnet has dependencies: terraform apply tfplan module.vpc.aws_route_table_association.private[2]: Destroying... [id=rtbassoc-051fbfbd50d9d8536] module.vpc.aws_route_table_association.private[2]: Destruction complete after 0s module.vpc.aws_subnet.private[2]: Destroying... [id=subnet-0710ac7c70335c514] module.vpc.aws_route_table.private[2]: Destroying... [id=rtb-09a5a0c0ac9ed26b2] module.vpc.aws_route_table.private[2]: Destruction complete after 1s module.vpc.aws_subnet.private[2]: Still destroying... [id=subnet-0710ac7c70335c514, 10s elapsed] ... Error: deleting EC2 Subnet (subnet-0710ac7c70335c514): DependencyViolation: The subnet 'subnet-0710ac7c70335c514' has dependencies and cannot be deleted. │ status code: 400, request id: 648b69ac-01eb-4435-a95c-ee2652a156d2

Additional context

Trying to delete the subnet from the console will say "Subnets have network interfaces and cannot be deleted". This is because an ENI still exists. I believe this occurs because terraform tries to delete the subnet before it runs this modification on the vpc_endpoint:

# module.vpc_endpoints.aws_vpc_endpoint.this["ec2messages"] will be updated in-place
  ~ resource "aws_vpc_endpoint" "this" {
        id                    = "vpce-07a71fd7508ad8fd8"
      ~ subnet_ids            = [
          - "subnet-0710ac7c70335c514",
            # (2 unchanged elements hidden)
        ]

[bryantbiggs]

I believe this occurs because terraform tries to delete the subnet before it runs this modification on the vpc_endpoint:

Correct. Its the order of operations that cause a chicken vs the egg type scenario:

Since the VPC endpoints module relies on the subnet IDs output from the VPC module, Terraform starts with the VPC module and attempts to remove the subnet. This subnet cannot be removed due to the fact that a endpoint is provisioned there

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.