Open jordanboston opened 11 months ago
Voting for Prioritization
Volunteering to Work on This Issue
Hey @jordanboston 👋 Thank you for taking the time to raise this! I took a look, and the schema for that resource hasn't changed since the resource was introduced, but perhaps something changed elsewhere. Are you able to supply a Terraform configuration that we can use to reproduce the issue, and/or trace logging?
vpc.tf
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "5.1.2"
name = "${var.env}-vpc"
cidr = var.cidr
azs = var.azs
public_subnets = var.public_subnets
private_subnets = var.private_subnets
public_subnet_assign_ipv6_address_on_creation = true
private_subnet_assign_ipv6_address_on_creation = true
public_subnet_enable_dns64 = false
private_subnet_enable_dns64 = false
public_subnet_ipv6_prefixes = var.public_subnets_ipv6_prefixes
private_subnet_ipv6_prefixes = var.private_subnets_ipv6_prefixes
enable_flow_log = true
flow_log_destination_type = "s3"
flow_log_destination_arn = module.s3_bucket.s3_bucket_arn
enable_nat_gateway = true
enable_ipv6 = true
manage_default_network_acl = false
manage_default_route_table = false
manage_default_security_group = false
# Added because it is currently false yet the default is true
public_subnet_enable_resource_name_dns_aaaa_record_on_launch = false
private_subnet_enable_resource_name_dns_aaaa_record_on_launch = false
# Added because it is currently true yet the default is false
map_public_ip_on_launch = true
# Specifying here to avoid a change on update
enable_dns_hostnames = false
}
module "tgw" {
source = "terraform-aws-modules/transit-gateway/aws"
version = "2.8.1"
name = "networking-tgw"
description = "Transit gateway shared with several other AWS accounts"
# When "true" there is no need for RAM resources if using multiple AWS accounts
enable_auto_accept_shared_attachments = true
enable_default_route_table_association = false
enable_default_route_table_propagation = false
ram_principals = var.organizations_accounts
tgw_default_route_table_tags = {
Purpose = "unused"
}
tags = {
Purpose = "tgw-complete-example"
}
}
resource "aws_ec2_transit_gateway_vpc_attachment" "networking_vpc_attachment" {
subnet_ids = module.vpc.private_subnets
transit_gateway_id = module.tgw.ec2_transit_gateway_id
vpc_id = module.vpc.vpc_id
ipv6_support = "enable"
transit_gateway_default_route_table_association = false
transit_gateway_default_route_table_propagation = false
# known issue with the continous state file change for this part
# https://github.com/hashicorp/terraform-provider-aws/issues/8383#issuecomment-680847938
lifecycle {
ignore_changes = [
transit_gateway_default_route_table_association,
transit_gateway_default_route_table_propagation
]
}
tags = {
Name = "${var.env}-tgw-attachment"
}
}
tgw.tf
data "aws_ec2_transit_gateway_vpc_attachments" "tgw_attachment" {}
locals {
attachment_ids = data.aws_ec2_transit_gateway_vpc_attachments.tgw_attachment.ids
excluded_attachment_ids = [
aws_ec2_transit_gateway_vpc_attachment.random_vpc_attachment.id,
data.terraform_remote_state.other_services_state_file.outputs.other_services_vpc_attachment_id
]
# Create a list of attachment IDs excluding the ones in the exclusion list
tgw_attachment_ids = [
for id in local.attachment_ids :
id if id != local.excluded_attachment_ids[0] && id != local.excluded_attachment_ids[1]
]
}
# Create the Transit Gateway Route Table for the shared-routes (all VPCs) routing
resource "aws_ec2_transit_gateway_route_table" "tgw_shared_route_table" {
transit_gateway_id = module.tgw.ec2_transit_gateway_id
tags = {
Name = "${var.env}-tgw-shared-route-table"
Description = "Shared route table for all VPCs"
}
}
# Create the Transit Gateway Route Table for the all-env-routes (all VPCs except production) routing
resource "aws_ec2_transit_gateway_route_table" "tgw_all_env_route_table" {
transit_gateway_id = module.tgw.ec2_transit_gateway_id
tags = {
Name = "${var.env}-tgw-all-env-route-table"
Description = "Shared route table that excludes the production VPCs"
}
}
######################## static routes for all-env to NAT gateway ##############################
resource "aws_ec2_transit_gateway_route" "tgw_route_internet_ipv4" {
destination_cidr_block = "0.0.0.0/0"
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.networking_vpc_attachment.id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_all_env_route_table.id
}
resource "aws_ec2_transit_gateway_route" "tgw_route_internet_ipv6" {
destination_cidr_block = "::/0"
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.networking_vpc_attachment.id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_all_env_route_table.id
}
#Flavius: disabled this because it's a "black-hole" and traffic stops
#resource "aws_ec2_transit_gateway_route" "blackhole_route_ipv4" {
# destination_cidr_block = var.accounts_vpc_cidr
# blackhole = true
# transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_all_env_route_table.id
#}
######################## static routes for shared to NAT gateway ##############################
resource "aws_ec2_transit_gateway_route" "tgw_route_shared_internet_ipv4" {
destination_cidr_block = "0.0.0.0/0"
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.networking_vpc_attachment.id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
resource "aws_ec2_transit_gateway_route" "tgw_route_shared_internet_ipv6" {
destination_cidr_block = "::/0"
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.networking_vpc_attachment.id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
######################## route table associations ######################################
resource "aws_ec2_transit_gateway_route_table_association" "tgw_other_vpcs_association" {
count = length(local.tgw_attachment_ids)
transit_gateway_attachment_id = local.tgw_attachment_ids[count.index]
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_all_env_route_table.id
}
resource "aws_ec2_transit_gateway_route_table_association" "tgw_networking_vpc_association" {
transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.networking_vpc_attachment.id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
resource "aws_ec2_transit_gateway_route_table_association" "tgw_shared_services_vpc_association" {
transit_gateway_attachment_id = data.terraform_remote_state.shared_services_state_file.outputs.shared_services_vpc_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
# Create the route table association for the VPN connection
resource "aws_ec2_transit_gateway_route_table_association" "tdc_vpn_route_table_association" {
transit_gateway_attachment_id = aws_vpn_connection.tdc_vpn_connection.transit_gateway_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
depends_on = [aws_vpn_connection.tdc_vpn_connection]
}
######################## route table propagation ######################################
resource "aws_ec2_transit_gateway_route_table_propagation" "tgw_other_vpcs_propagation" {
count = length(local.tgw_attachment_ids)
transit_gateway_attachment_id = local.tgw_attachment_ids[count.index]
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
resource "aws_ec2_transit_gateway_route_table_propagation" "tgw_shared_services_vpc_propagation" {
transit_gateway_attachment_id = data.terraform_remote_state.shared_services_state_file.outputs.shared_services_vpc_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_all_env_route_table.id
}
resource "aws_ec2_transit_gateway_route_table_propagation" "tgw_shared_services_networking_vpc_propagation" {
transit_gateway_attachment_id = data.terraform_remote_state.shared_services_state_file.outputs.shared_services_vpc_attachment_id
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
}
# route from all other VPCs cidr ranges to the tgw in the networking account through the public subnets
resource "aws_route" "public_route_from_vpcs_to_tgw" {
destination_cidr_block = var.accounts_vpc_cidr
route_table_id = module.vpc.public_route_table_ids[0]
transit_gateway_id = module.tgw.ec2_transit_gateway_id
}
# Create the route for the VPN connection
resource "aws_ec2_transit_gateway_route" "tdc_vpn_route" {
count = length(var.ipsec_onprem_destination_subnets)
destination_cidr_block = var.ipsec_onprem_destination_subnets[count.index]
transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.tgw_shared_route_table.id
transit_gateway_attachment_id = aws_vpn_connection.tdc_vpn_connection.transit_gateway_attachment_id
blackhole = false
depends_on = [aws_vpn_connection.tdc_vpn_connection]
}
I've also been seeing strange behavior now. Even after bumping back up to provider 5.21.0 I got output showing: (id's removed)
Terraform will perform the following actions:
# aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[4] will be created
+ resource \"aws_ec2_transit_gateway_route_table_association\" \"tgw_other_vpcs_association\" {
+ id = (known after apply)
+ replace_existing_association = false
+ resource_id = (known after apply)
+ resource_type = (known after apply)
+ transit_gateway_attachment_id = \"tgw-attach-exists-already!\"
+ transit_gateway_route_table_id = \"tgw-rtb-redacted\"
}
Plan: 1 to add, 0 to change, 0 to destroy
But then running again later with no changes getting all these replacements when running a plan.
Plan: 8 to add, 0 to change, 7 to destroy.
I'm pretty confused right now as to what is actually going on. 😆
There are some other things like trying to create a resource that already has the id associated which causes an error, but I don't want to add to much more noise here. Thanks
Here is some log output from Github Actions. Not sure how much of this should be posted publicly, so I've removed a lot of the identifiers in it.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/home/runner/work/_temp/b15e3028-1959-4c29-ac58-acf936ae8be2/terraform-bin workspace select networking
Switched to workspace "networking".
Run cd ${environment}
/home/runner/work/_temp/b15e3028-1959-4c29-ac58-acf936ae8be2/terraform-bin apply -input=false tf.plan/tf.plan
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[1]: Destroying... [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Destroying... [id=tgw-rtb-064...._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[3]: Destroying... [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[2]: Destroying... [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[0]: Destroying... [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Destroying... [id=tgw-rtb-064...._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Destroying... [id=tgw-rtb-064...._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Destroying... [id=tgw-rtb-064...._tgw-attach-00...]
module.s3_bucket.aws_s3_bucket_public_access_block.this[0]: Modifying... [id=networking-vpc-flow-logs-to-s3]
module.security_group_networking.aws_security_group.this_name_prefix[0]: Modifying... [id=sg-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[3]: Destruction complete after 0s
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[3]: Creating...
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[0]: Destruction complete after 0s
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[4]: Creating...
module.security_group_networking.aws_security_group.this_name_prefix[0]: Modifications complete after 0s [id=sg-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[4]: Creating...
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[1]: Destruction complete after 0s
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[2]: Destruction complete after 0s
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[0]: Creating...
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[1]: Creating...
module.s3_bucket.aws_s3_bucket_public_access_block.this[0]: Modifications complete after 0s [id=networking-vpc-flow-logs-to-s3]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[2]: Creating...
module.ec2_instance_networking.aws_instance.this[0]: Modifying... [id=i-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[3]: Creation complete after 1s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[1]: Creation complete after 1s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[4]: Creation complete after 1s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[0]: Creation complete after 1s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_propagation.tgw_other_vpcs_propagation[2]: Creation complete after 1s [id=tgw-rtb-00..._tgw-attach-00...]
module.ec2_instance_networking.aws_instance.this[0]: Modifications complete after 2s [id=i-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Still destroying... [id=tgw-rtb-064...._tgw-attach-089..., 10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Still destroying... [id=tgw-rtb-064...._tgw-attach-089..., 20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Still destroying... [id=tgw-rtb-064...._tgw-attach-089..., 30s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 30s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 30s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Still destroying... [id=tgw-rtb-064...._tgw-attach-00..., 30s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Destruction complete after 34s
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Destruction complete after 34s
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Destruction complete after 34s
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Destruction complete after 34s
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Creating...
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Creating...
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Creating...
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Creating...
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Still creating... [10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Still creating... [10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Still creating... [10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Still creating... [10s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Still creating... [20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Still creating... [20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Still creating... [20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Still creating... [20s elapsed]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[3]: Creation complete after 24s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[1]: Creation complete after 25s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[2]: Creation complete after 25s [id=tgw-rtb-00..._tgw-attach-00...]
aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[0]: Creation complete after 25s [id=tgw-rtb-00..._tgw-attach-00...]
╷
│ Error: creating EC2 Transit Gateway Route Table Association (tgw-rtb-064..._tgw-attach-00...): Resource.AlreadyAssociated: Transit Gateway Attachment tgw-attach-00... is already associated to a route table.
│ status code: 400, request id: 10979fc2-43b8-46b7-a131-11fff7e2fc32
│
│ with aws_ec2_transit_gateway_route_table_association.tgw_other_vpcs_association[4],
│ on tgw_routing.tf line 73, in resource "aws_ec2_transit_gateway_route_table_association" "tgw_other_vpcs_association":
│ 73: resource "aws_ec2_transit_gateway_route_table_association" "tgw_other_vpcs_association" {
│
╵
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.
The existing association can be seen in both the AWS Console and the statefile. So why it wants to create it is confusing here.
At this point I'm not sure if this is a bug but there is strange stuff going on surrounding this, that we had not seen before.
@jordanboston are you still seeing this issue?
Terraform Core Version
1.6.1
AWS Provider Version
5.21.0
Affected Resource(s)
Resources here are being replaced when using version above v5.19.0 It does not make sense why these are getting replaced with the version bump.
I'm not positive that this is a bug, but it seems like it might be.
It's showing that the id is what forces replacement, but these have not changed. What we have in the statefile matches what is set here before the change, and when going back to 5.19 the issue is not there.
EXAMPLE:
~ transit_gateway_attachment_id = "tgw-attach-0a1b2c3d4e5f6g7h8i9j" -> "tgw-attach-0k1l2m3n4o5p6q7r8s9t" # forces replacement
Same as what was reported here: https://github.com/hashicorp/terraform-provider-aws/issues/32195
Expected Behavior
Nothing changes in TGW whatsoever.
Actual Behavior
Resources are destroyed and new ones added.
Relevant Error/Panic Output Snippet
Terraform Configuration Files
None are needed. It does not happen in version 5.19.0 with the same exact config. Only when bumping up versions, with no changes to config at all.
Steps to Reproduce
Bump version from anything above 5.19.0.
Debug Output
No response
Panic Output
No response
Important Factoids
Originally posted here: https://github.com/hashicorp/terraform-provider-aws/issues/32195#issue-1771784171 In version provider registry.terraform.io/hashicorp/aws v5.5.0
This issue seems to arise when moving past v5.19 > When moving back to to 5.19 the issue does not seem present.
References
https://github.com/hashicorp/terraform-provider-aws/issues/32195#issue-1771784171
Would you like to implement a fix?
None