search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.85k stars 9.19k forks source link

[Bug]: Not all resources are destroyed after removal their configuration from TF #37045

Open EugenKon opened 7 months ago

EugenKon commented 7 months ago

Terraform Core Version

v1.7.4

AWS Provider Version

v4.67.0

Affected Resource(s)

Expected Behavior

The added route should be physically removed when it was removed from TF configuration.

Actual Behavior

The route was not removed: image

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

For this TF configuration:

--- a/aws/network.tf
+++ b/aws/network.tf
@@ -101,6 +101,25 @@ resource "aws_route_table" "prd_plntr" {
   }
 }

+resource "aws_eip" "nat-prd_plntr" {
+  tags = {
+    Name = "Public IP for plntr-nat-gateway"
+  }
+}
+
+resource "aws_nat_gateway" "plntr-nat-gateway" {
+  allocation_id = aws_eip.nat-prd_plntr.id
+  subnet_id     = aws_subnet.a_prd_plntr.id
+
+  tags = {
+    Name = "NAT Gateway prd_plntr"
+  }
+
+  depends_on = [aws_internet_gateway.prd_plntr]
+}
+
 resource "aws_route_table" "private_prd_plntr" {
   vpc_id = aws_vpc.prd_plntr.id

+  # Route for private network to NAT Gateway
+  route {
+    cidr_block     = "0.0.0.0/0"
+    nat_gateway_id = aws_nat_gateway.plntr-nat-gateway.id
+  }
+
+
   tags = {
     Name = "private-prd.plntr.ca"
   }

Steps to Reproduce

This TF plan was generated: terraform plan

TF config ```tf Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create ~ update in-place Terraform will perform the following actions: # aws_eip.nat-prd_plntr will be created + resource "aws_eip" "nat-prd_plntr" { + allocation_id = (known after apply) + association_id = (known after apply) + carrier_ip = (known after apply) + customer_owned_ip = (known after apply) + domain = (known after apply) + id = (known after apply) + instance = (known after apply) + network_border_group = (known after apply) + network_interface = (known after apply) + private_dns = (known after apply) + private_ip = (known after apply) + public_dns = (known after apply) + public_ip = (known after apply) + public_ipv4_pool = (known after apply) + tags = { + "Name" = "Public IP for plntr-nat-gateway" } + tags_all = { + "Name" = "Public IP for plntr-nat-gateway" } + vpc = (known after apply) } # aws_nat_gateway.plntr-nat-gateway will be created + resource "aws_nat_gateway" "plntr-nat-gateway" { + allocation_id = (known after apply) + association_id = (known after apply) + connectivity_type = "public" + id = (known after apply) + network_interface_id = (known after apply) + private_ip = (known after apply) + public_ip = (known after apply) + subnet_id = "subnet-***" + tags = { + "Name" = "NAT Gateway prd_plntr" } + tags_all = { + "Name" = "NAT Gateway prd_plntr" } } # aws_route_table.private_prd_plntr will be updated in-place ~ resource "aws_route_table" "private_prd_plntr" { id = "rtb-***" ~ route = [ + { + carrier_gateway_id = "" + cidr_block = "0.0.0.0/0" + core_network_arn = "" + destination_prefix_list_id = "" + egress_only_gateway_id = "" + gateway_id = "" + instance_id = "" + ipv6_cidr_block = "" + local_gateway_id = "" + nat_gateway_id = (known after apply) + network_interface_id = "" + transit_gateway_id = "" + vpc_endpoint_id = "" + vpc_peering_connection_id = "" }, ] tags = { "Name" = "private-prd.plntr.ca" } # (5 unchanged attributes hidden) } Plan: 2 to add, 1 to change, 0 to destroy. ```

But when I applied this plan, removed resorces from configuration and generate plan, then not all changes resources were returned to the source state.

TF plan ```tf Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # aws_eip.nat-prd_plntr will be destroyed # (because aws_eip.nat-prd_plntr is not in configuration) - resource "aws_eip" "nat-prd_plntr" { - allocation_id = "eipalloc-09a3***" -> null - association_id = "eipassoc-0486***" -> null - domain = "vpc" -> null - id = "eipalloc-09a3***" -> null - network_border_group = "*" -> null - network_interface = "eni-0306*" -> null - private_dns = "ip-*.ec2.internal" -> null - private_ip = "*" -> null - public_dns = "ec2-*.compute-1.amazonaws.com" -> null - public_ip = "*" -> null - public_ipv4_pool = "amazon" -> null - tags = { - "Name" = "Public IP for plntr-nat-gateway" } -> null - tags_all = { - "Name" = "Public IP for plntr-nat-gateway" } -> null - vpc = true -> null } # aws_nat_gateway.plntr-nat-gateway will be destroyed # (because aws_nat_gateway.plntr-nat-gateway is not in configuration) - resource "aws_nat_gateway" "plntr-nat-gateway" { - allocation_id = "eipalloc-09a3*" -> null - association_id = "eipassoc-0486*" -> null - connectivity_type = "public" -> null - id = "nat-0a83*" -> null - network_interface_id = "eni-0306*" -> null - private_ip = "*" -> null - public_ip = "*" -> null - subnet_id = "subnet-*" -> null - tags = { - "Name" = "NAT Gateway prd_plntr" } -> null - tags_all = { - "Name" = "NAT Gateway prd_plntr" } -> null } Plan: 0 to add, 0 to change, 2 to destroy. ```

Here you can see that the default route was not deleted.

TF state ```diff --- a/aws/terraform.tfstate +++ b/aws/terraform.tfstate @@ -1,7 +1,7 @@ { "version": 4, "terraform_version": "1.7.4", - "serial": 770, + "serial": 773, "lineage": "*", "outputs": { "db_a_addr": { @@ -2168,46 +2168,6 @@ } ] }, - { - "mode": "managed", - "type": "aws_eip", - "name": "nat-prd_plntr", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "address": null, - "allocation_id": "eipalloc-09a3*", - "associate_with_private_ip": null, - "association_id": "", - "carrier_ip": "", - "customer_owned_ip": "", - "customer_owned_ipv4_pool": "", - "domain": "vpc", - "id": "eipalloc-09a3*", - "instance": "", - "network_border_group": "*", - "network_interface": "", - "private_dns": null, - "private_ip": "", - "public_dns": "ec2-*.compute-1.amazonaws.com", - "public_ip": "*", - "public_ipv4_pool": "amazon", - "tags": { - "Name": "Public IP for plntr-nat-gateway" - }, - "tags_all": { - "Name": "Public IP for plntr-nat-gateway" - }, - "timeouts": null, - "vpc": true - }, - "sensitive_attributes": [], - "private": "*> - } - ] - }, { "mode": "managed", "type": "aws_iam_group", @@ -6257,41 +6217,6 @@ } ] }, - { - "mode": "managed", - "type": "aws_nat_gateway", - "name": "plntr-nat-gateway", - "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]", - "instances": [ - { - "schema_version": 0, - "attributes": { - "allocation_id": "eipalloc-09a3*", - "association_id": "eipassoc-0486*", - "connectivity_type": "public", - "id": "nat-0a83*", - "network_interface_id": "eni-0306*", - "private_ip": "*", - "public_ip": "*", - "subnet_id": "subnet-*", - "tags": { - "Name": "NAT Gateway prd_plntr" - }, - "tags_all": { - "Name": "NAT Gateway prd_plntr" - } - }, - "sensitive_attributes": [], - "private": "bnVsbA==", - "dependencies": [ - "aws_eip.nat-prd_plntr", - "aws_internet_gateway.prd_plntr", - "aws_subnet.a_prd_plntr", - "aws_vpc.prd_plntr" - ] - } - ] - }, { "mode": "managed", "type": "aws_route53_record", @@ -8748,10 +8673,6 @@ "sensitive_attributes": [], "private": "*"> "dependencies": [ - "aws_eip.nat-prd_plntr", - "aws_internet_gateway.prd_plntr", - "aws_nat_gateway.plntr-nat-gateway", - "aws_subnet.a_prd_plntr", "aws_vpc.prd_plntr" ] } ```

For the above state you can see that aws_nat_gateway and aws_eip resources were removed from state, but route of aws_route_table is still lurking in the state:

image

Even after manual route removal via AWS Console, TF does not see the change.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 7 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

Michagogo commented 6 months ago

As far as I can tell you don’t provide the full configuration here - there any other route{} blocks in the route table in question? If not, what’s happening here might be that when there are no routes defined inline the route table resource doesn’t manage the routes for the table at all, to allow for external management (e.g. via route resources).

In that case, I believe setting route = [] would have the desired affect, and remove all routes from the table.

I do agree that the behavior is confusing - once a route table (or any other resource that has a similar format with subresources) has its set of those sub resources managed by TF, unmanaging that set by emptying it shouldn’t leave the resources in place.

EugenKon commented 6 months ago

Here is my full configuration for that section (I updated my first post):

resource "aws_route_table" "private_prd_plntr" {
  vpc_id = aws_vpc.prd_plntr.id

  /* The default route, mapping the VPC's CIDR block to "local",
   * created implicitly and cannot be specified.
   * No more routes to be defined.
   */

  tags = {
    Name = "private-prd.plntr.ca"
  }
}

Yeah, probably route=[] should be the value by default in the Terraform.

Michagogo commented 6 months ago

Yeah, probably route=[] should be the value by default in the Terraform.

No, it definitely shouldn’t be the default — the default needs to be for the resource not to manage it at all, in order to allow creation of routes either in their own separate resources or outside TF entirely. The bug here is the specific edge case of removing all routes from the resource, which currently just unmanages the route list entirely, leaving orphaned routes that TF created but no longer knows about to destroy. IMO it should indeed remove those routes, after which the route list should go back to being unmanaged.