search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.77k stars 9.12k forks source link

[Bug]: AcceptTransitGatewayPeeringAttachment inside same region and same account not working #38027

Closed narimantos closed 1 month ago

narimantos commented 3 months ago

Terraform Core Version

1.8.5

AWS Provider Version

5.54.1

Affected Resource(s)

Hi I think i have encountered a bug with the aws_ec2_transit_gateway_peering_attachment and aws_ec2_transit_gateway_peering_attachment_accepter resources.

I have 3 VPC's

My provider is the same account but different region: aws.UK = eu-west-2 aws.EU = eu-central-1

Expected Behavior

resource "aws_ec2_transit_gateway_peering_attachment_accepter" "generic_eu" {
  provider                      = aws.eu
  transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.generic_eu.id
}

Should work.

Actual Behavior

cannot accepts the tgw-attach-XXXXX as source of the peering request if its in the same region.

Relevant Error/Panic Output Snippet

> │ Error: accepting EC2 Transit Gateway Peering Attachment (tgw-attach-0dffb49e9149ee78f): operation error EC2: AcceptTransitGatewayPeeringAttachment, https response error StatusCode: 400, RequestID: 354a8b21-6b87-4aa7-b3d9-40081216e0ce, api error InvalidParameterValue: Cannot accept tgw-attach-0dffb49e9149ee78f as the source of the peering request.
│ 
│   with aws_ec2_transit_gateway_peering_attachment_accepter.generic_eu,
│   on 00-generic-01-vpc.tf line 107, in resource "aws_ec2_transit_gateway_peering_attachment_accepter" "generic_eu":
│  107: resource "aws_ec2_transit_gateway_peering_attachment_accepter" "generic_eu" {
│

Terraform Configuration Files

resource "aws_ec2_transit_gateway" "generic_tgw" {
  provider                        = aws.eu
  amazon_side_asn                 = 64512
  auto_accept_shared_attachments  = "disable"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  description                     = "generic"
  dns_support                     = "enable"
  vpn_ecmp_support                = "enable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "generic_main" {
  provider = aws.eu
  subnet_ids = [
    aws_subnet.generic_mgmt_a_private.id,
    # aws_subnet.generic_mgmt_a_public.id
  ]
  transit_gateway_id                              = aws_ec2_transit_gateway.generic_tgw.id
  vpc_id                                          = aws_vpc.generic_vpc.id
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

resource "aws_ec2_transit_gateway" "eu_main" {
  provider                        = aws.eu
  amazon_side_asn                 = 64512
  auto_accept_shared_attachments  = "disable"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  description                     = "main"
  dns_support                     = "enable"
  vpn_ecmp_support                = "enable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "eu_main" {
  provider = aws.eu
  subnet_ids = [
    aws_subnet.eu_a.id,
    aws_subnet.eu_b.id,
    aws_subnet.eu_c.id
  ]
  transit_gateway_id                              = aws_ec2_transit_gateway.eu_main.id
  vpc_id                                          = aws_vpc.eu_main.id
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

resource "aws_ec2_transit_gateway" "uk_main" {
  provider                        = aws.uk
  amazon_side_asn                 = 64512
  auto_accept_shared_attachments  = "disable"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  description                     = "main"
  dns_support                     = "enable"
  vpn_ecmp_support                = "enable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "uk_main" {
  provider = aws.uk
  subnet_ids = [
    aws_subnet.uk_a.id,
    aws_subnet.uk_b.id,
    aws_subnet.uk_c.id
  ]
  transit_gateway_id                    = aws_ec2_transit_gateway.uk_main.id
  vpc_id                                          = aws_vpc.uk_main.id
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

resource "aws_ec2_transit_gateway_peering_attachment" "generic_eu" {
  provider                = aws.eu
  peer_region             = var.eu_aws_region
  peer_transit_gateway_id = aws_ec2_transit_gateway.eu_main.id
  transit_gateway_id      = aws_ec2_transit_gateway.generic_tgw.id
}

resource "aws_ec2_transit_gateway_peering_attachment" "generic_uk" {
  provider                = aws.eu
  peer_region             = var.uk_aws_region
  peer_transit_gateway_id = aws_ec2_transit_gateway.uk_main.id
  transit_gateway_id      = aws_ec2_transit_gateway.generic_tgw.id
}

## this resource fails
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "generic_eu" {
  provider                      = aws.eu
  transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.generic_eu.id
}

## this resource works
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "generic_peer_uk" {
  provider                      = aws.uk
  transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.generic_uk.id
}

Steps to Reproduce

terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No

github-actions[bot] commented 3 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

uplight-james commented 3 months ago

I plan to let our TAM know about this as well but in AWS I am noticing some interesting behavior. Our use case is multi-account, we are creating a TGW & peering to another - both in us-east-1, one in account A, one in account B.

When we create the TGWs and peering attachments (not yet accepted), we have no problems in TF. When we try to accept the peering attachment, we get the error mentioned in this message.

The TGW attachments are visible in both account A and account b, if your attaching a TGW in one region to a TGW in a different region, the id for this resource is the same.

HOWEVER, if you're attaching a TGW in one region to another TGW in the SAME region the id is completely different (at least in different accounts, not sure if this is true when they're in the same account).

narimantos commented 3 months ago

Thanks for the comment @uplight-james. Please update (comment in) this issue if you have any updates from TAM.

o6uoq commented 3 months ago

The TGW attachments are visible in both account A and account b, if your attaching a TGW in one region to a TGW in a different region, the id for this resource is the same.

HOWEVER, if you're attaching a TGW in one region to another TGW in the SAME region the id is completely different (at least in different accounts, not sure if this is true when they're in the same account).

@uplight-james is this related to https://github.com/hashicorp/terraform-provider-aws/issues/24677 and https://github.com/hashicorp/terraform-provider-aws/pull/36761 ?

o6uoq commented 2 months ago

The TGW attachments are visible in both account A and account b, if your attaching a TGW in one region to a TGW in a different region, the id for this resource is the same.

@uplight-james have you been able to demonstrate this outside of Terraform i.e. via AWS CLI/API?

uplight-james commented 2 months ago

Thanks for the comment @uplight-james. Please update (comment in) this issue if you have any updates from TAM.

@narimantos for sure, this is annoying for me to so I'll continue to check in :) He was able to confirm that there is an ID constraint for TGW attachments, all TGW attachments MUST have unique IDs within the same region regardless of the account/accounts containing the attachment.

@uplight-james is this related to #24677 and #36761

@o6uoq Yep! That workaround w/ the data block is required when peering TGWs within the same region.

@uplight-james have you been able to demonstrate this outside of Terraform i.e. via AWS CLI/API?

@o6uoq I didn't try w/ the CLI but observed that each side of the attachment has a differing ID within the same region and my TAM confirmed that unique ID constraint. I can try to whip up a CLI example in the next week, workload allowing, but no guarantees.

o6uoq commented 2 months ago

@uplight-james we have implemented the workaround, which was then confirmed via the official HashiCorp AWS Provider docs: https://github.com/hashicorp/terraform-provider-aws/pull/36761/files

We have bumped our provider to latest i.e. greater than v5.45

We are still seeing data refreshes, triggering a resource recreaton. The issue can be recreated when bumping git_ref, even if there's no changes between the branches/tags.

I feel this bug/issue may be related, but requires additional time to deep dive and investigate: https://github.com/hashicorp/terraform-provider-aws/issues/29421

You can use something like the below for AWS CLI, however it would be good to know exactly which ID(s) you are referring to? As you can see here and below, there's more than one!

aws ec2 describe-transit-gateway-peering-attachments \
  --profile $AWS_PROFILE \
  --region $AWS_REGOIN \
  --query 'TransitGatewayPeeringAttachments[*].{TransitGatewayAttachmentId:TransitGatewayAttachmentId, State:State, RequesterTgwId:RequesterTgwInfo.TransitGatewayId, RequesterRegion:RequesterTgwInfo.Region, RequesterOwnerId:RequesterTgwInfo.OwnerId, AccepterTgwId:AccepterTgwInfo.TransitGatewayId, AccepterRegion:AccepterTgwInfo.Region, AccepterOwnerId:AccepterTgwInfo.OwnerId, AccepterTransitGatewayAttachmentId:AccepterTgwInfo.TransitGatewayAttachmentId}' \
  --output table

@uplight-james if you let me know which ID(s) I should be referring to, happy to put something together via AWS CLI. I'm assuming I'm going to have to run this against one AWS Region, then another, and compare IDs - I just want to know which IDs exactly!

uplight-james commented 2 months ago

@o6uoq can you confirm the recreated resources is https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_transit_gateway_peering_attachment_accepter?

If you can share a sanitized plan or sample of your TF code I might be able to comment further.

The specific ID causing this problem is transit_gateway_attachment_id (tf), when you create a TGW in the same region but different accounts, the attachment is visible in both accounts but will have different IDs in each account.

justinretzolk commented 1 month ago

Hi all 👋 After reviewing the discussion here, it looks like this came down to a need for a slight configuration change. With that in mind, I'm going to close this issue. If you encounter any further issues with the provider, please do let us know!

github-actions[bot] commented 1 month ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

github-actions[bot] commented 6 days ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.