[Bug]: Inconsistent plan regarding the creation of tgw attachment route table association

[laserpedro]

Terraform Core Version

1.9.5

AWS Provider Version

5.61.0

Affected Resource(s)

• aws_ec2_transit_gateway_route_table_association
• aws_ec2_transit_gateway_vpc_attachment

Expected Behavior

Target: create a vpc tgw attachment to a regional tgw. Associate the tgw attachment to an inspection route table / propagate the tgw attachment to a spoke route table.

Plan at tf apply:

  # module.core_infra.aws_ec2_transit_gateway_route_table_association.route_table_association[0] will be created
  + resource "aws_ec2_transit_gateway_route_table_association" "route_table_association" {
      + id                             = (known after apply)
      + replace_existing_association   = false
      + resource_id                    = (known after apply)
      + resource_type                  = (known after apply)
      + transit_gateway_attachment_id  = (known after apply)
      + transit_gateway_route_table_id = "tgw-rtb-inspection" # NON DEFAULT TGW RT
    }

  # module.core_infra.aws_ec2_transit_gateway_route_table_propagation.route_table_propagation[0] will be created
  + resource "aws_ec2_transit_gateway_route_table_propagation" "route_table_propagation" {
      + id                             = (known after apply)
      + resource_id                    = (known after apply)
      + resource_type                  = (known after apply)
      + transit_gateway_attachment_id  = (known after apply)
      + transit_gateway_route_table_id = "tgw-rtb-spoke" # NON DEFAULT TGW RT
    }

  # module.core_infra.aws_ec2_transit_gateway_vpc_attachment.vpc_attachment[0] will be created
  + resource "aws_ec2_transit_gateway_vpc_attachment" "vpc_attachment" {
      + appliance_mode_support                          = "enable"
      + dns_support                                     = "enable"
      + id                                              = (known after apply)
      + ipv6_support                                    = "disable"
      + subnet_ids                                      = [
          + "subnet-1",
          + "subnet-2",
          + "subnet-3",
        ]
      + transit_gateway_default_route_table_association = (known after apply)
      + transit_gateway_default_route_table_propagation = (known after apply)
      + transit_gateway_id                              = "tgw-regional-tgw"
      + vpc_id                                          = "vpc-123456"
      + vpc_owner_id                                    = (known after apply)
    }

Actual Behavior

• tgw attachment association created on the default route table though / provider failed due to conflicting resources as it tries to re-create the association of the non-default route table in the default one ....

│ Error: creating EC2 Transit Gateway Route Table Association (tgw-rtb-inspection_tgw-attach-78910): operation error EC2: AssociateTransitGatewayRouteTable, https response error StatusCode: 400, RequestID: f5a087d3-f8bb-4631-beeb-3c2d2d73e52e, api error Resource.AlreadyAssociated: Transit Gateway Attachment tgw-attach-1234 is already associated to a route table.
│ 
│   with module.core_infra.aws_ec2_transit_gateway_route_table_association.route_table_association[0],
│   on .terraform/modules/core_infra/modules/core-infra/main.tf line 223, in resource "aws_ec2_transit_gateway_route_table_association" "route_table_association":
│  223: resource "aws_ec2_transit_gateway_route_table_association" "route_table_association" {
│  

Relevant Error/Panic Output Snippet

│ Error: creating EC2 Transit Gateway Route Table Association (tgw-rtb-inspection_tgw-attach-78910): operation error EC2: AssociateTransitGatewayRouteTable, https response error StatusCode: 400, RequestID: f5a087d3-f8bb-4631-beeb-3c2d2d73e52e, api error Resource.AlreadyAssociated: Transit Gateway Attachment tgw-attach-1234 is already associated to a route table.
│ 
│   with module.core_infra.aws_ec2_transit_gateway_route_table_association.route_table_association[0],
│   on .terraform/modules/core_infra/modules/core-infra/main.tf line 223, in resource "aws_ec2_transit_gateway_route_table_association" "route_table_association":
│  223: resource "aws_ec2_transit_gateway_route_table_association" "route_table_association" {
│

IMPORTANT: once the vpc tgw attachment association to the tgw default route table is deleted we can apply and create the tgw vpc attachment association to the inspection route table as displayed in the original tf plan.

Terraform Configuration Files

resource "aws_ec2_transit_gateway_vpc_attachment" "vpc_attachment" {
  count              = var.create_tgw_attachment ? 1 : 0
  transit_gateway_id = var.tgw_id
  vpc_id             = module.vpc.vpc_id

  subnet_ids = module.vpc.public_subnets

  dns_support            = "enable"
  ipv6_support           = "disable"
  appliance_mode_support = "enable"

  depends_on = [module.vpc]

  tags = {
    Name = var.cluster_name
  }
}

# NOTE: due to firewall setup we need to associate to one route table and to propagate to another
resource "aws_ec2_transit_gateway_route_table_association" "route_table_association" {
  provider                       = aws.network
  count                          = var.create_tgw_attachment ? 1 : 0
  transit_gateway_attachment_id  = resource.aws_ec2_transit_gateway_vpc_attachment.vpc_attachment[count.index].id
  transit_gateway_route_table_id = var.transit_gateway_association_route_table_id
}

resource "aws_ec2_transit_gateway_route_table_propagation" "route_table_propagation" {
  provider                       = aws.network
  count                          = var.create_tgw_attachment ? 1 : 0
  transit_gateway_attachment_id  = resource.aws_ec2_transit_gateway_vpc_attachment.vpc_attachment[count.index].id
  transit_gateway_route_table_id = var.transit_gateway_propagation_route_table_id
}

Steps to Reproduce

create a tgw with a 2 non default route tables and apply the tf configuration above with hard coded values for the association route table (the inspection route table) and for the propagation route table (the spoke route table).

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @laserpedro 👋 Thank you for taking the time to raise this! It looks like this has come up before in #16452, where a couple of workarounds were offered. Can you review that report (and/or the PR that closed it, #31452) and let us know if the options discussed there resolve the issue for you?

[laserpedro]

Hey @justinretzolk thank you very much for your assistance on this: I used the PR https://github.com/hashicorp/terraform-provider-aws/pull/31452 and it performed exactly what I wanted. The vpc tgw attachment is no longer associated by default to the default tgw route tables. However, by reading at the doc for the tgw vpc attachment:

transit_gateway_default_route_table_association - (Optional) Boolean whether the VPC Attachment should be associated with the EC2 Transit Gateway association default route table. This cannot be configured or perform drift detection with Resource Access Manager shared EC2 Transit Gateways. Default value: true. transit_gateway_default_route_table_propagation - (Optional) Boolean whether the VPC Attachment should propagate routes with the EC2 Transit Gateway propagation default route table. This cannot be configured or perform drift detection with Resource Access Manager shared EC2 Transit Gateways. Default value: true.

I am bit confused: I let those parameters to their default value and declare the route table association this way:

resource "aws_ec2_transit_gateway_route_table_association" "route_table_association" {
  provider                       = aws.network
  count                          = var.create_tgw_attachment ? 1 : 0
  transit_gateway_attachment_id  = resource.aws_ec2_transit_gateway_vpc_attachment.vpc_attachment[count.index].id
  transit_gateway_route_table_id = var.transit_gateway_association_route_table_id
  replace_existing_association   = true
} 

Originally I thought that those two arguments would have been enough to prevent the association to the default route table. From the route table association documentation:

replace_existing_association - (Optional) Boolean whether the Gateway Attachment should remove any current Route Table association before associating with the specified Route Table. Default value: false. This argument is intended for use with EC2 Transit Gateways shared into the current account, otherwise the transit_gateway_default_route_table_association argument of the aws_ec2_transit_gateway_vpc_attachment resource should be used.

My understanding is that the transit_gateway_default_route_table_propagation and transit_gateway_default_route_table_association set to false work if the tgw is in the same account and NOT shared. if the tgw is shared then there are not relevant and we need to use the argument replace_existing_association and in my case set it to false to prevent the auto attachment ?