examples/transit-gateway-cross-account-peering-attachment is flawed

pmcevoy commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.1.9
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v4.12.1

Affected Resource(s)

aws_ec2_transit_gateway_peering_attachment
aws_ec2_transit_gateway_peering_attachment_accepter

Terraform Configuration Files

Note that the following is adapted from https://github.com/hashicorp/terraform-provider-aws/blob/main/examples/transit-gateway-cross-account-peering-attachment/main.tf but using assume_role and "data" resources to get access to the TGWs that were created in other TF files

# First accepts the Peering attachment.
provider "aws" {
    alias = "first"
}

# Second creates the Peering attachment.
provider "aws" {
    alias = "second"

    assume_role {
        role_arn     = "arn:aws:iam::11111122222233333:role/DingTgwPeeringCreator"
        session_name = "tgw-peering"
        external_id  = "TgwCreation"
    }
}

data "aws_caller_identity" "first" {
    provider = aws.first
}

data "aws_ec2_transit_gateway" "first" {
    provider = aws.first

    filter {
        name  = "tag:Name"
        values = [ "private" ]
    }
    filter {
        name  = "state"
        values = ["available"]
    }
}

data "aws_ec2_transit_gateway" "second" {
    provider = aws.second

    filter {
        name  = "tag:Name"
        values = [ "DingDev" ]
    }
    filter {
        name  = "state"
        values = ["available"]
    }
}

# Create the Peering attachment in the second account...
resource "aws_ec2_transit_gateway_peering_attachment" "example" {
    provider = aws.second

    peer_account_id         = data.aws_caller_identity.first.account_id
    peer_region             = "eu-west-1"
    peer_transit_gateway_id = data.aws_ec2_transit_gateway.first.id
    transit_gateway_id      = data.aws_ec2_transit_gateway.second.id
    tags = {
        Name = "terraform-example"
        Side = "Creator"
    }
}

# ...and accept it in the first account.
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example" {
    provider = aws.first

    transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.example.id
    tags = {
        Name = "terraform-example"
        Side = "Acceptor"
    }
}

Debug Output

Panic Output

Expected Behavior

Peering request to be accepted

Actual Behavior

│ Error: error accepting EC2 Transit Gateway Peering Attachment: InvalidParameterValue: Cannot accept tgw-attach-059233eb3472ea83b as the source of the peering request.
│       status code: 400, request id: d2c2c1fa-0911-45d9-a3f2-8e5464773404
│
│   with aws_ec2_transit_gateway_peering_attachment_accepter.example,
│   on main.tf line 73, in resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example":
│   73: resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example" {
│
╵

Steps to Reproduce

Following the example listed here https://github.com/hashicorp/terraform-provider-aws/blob/main/examples/transit-gateway-cross-account-peering-attachment/main.tf

Important Factoids

I found a comment on another issue that indicated that two attchments are actually created - one in the first and one in the second account and that we need to accept using the id of the attachment in "first" account (using the aliases from the example).

https://github.com/hashicorp/terraform-provider-aws/issues/23828#issuecomment-1083725508

References

23828

Work around

Per the comment in https://github.com/hashicorp/terraform-provider-aws/issues/23828#issuecomment-1083725508, the example can be made to work by adding the following "data":

data "aws_ec2_transit_gateway_peering_attachment" "fix" {
    provider = aws.first
    depends_on = [aws_ec2_transit_gateway_peering_attachment.example]
    filter {
        name   = "transit-gateway-id"
        values = [aws_ec2_transit_gateway_peering_attachment.example.peer_transit_gateway_id]
    }
    filter {
        name   = "state"
        values = ["pendingAcceptance"]
    }
}

and updating the accepter to use the attachment ID of the "fix" attachment

# ...and accept it in the first account.
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example" {
    provider = aws.first

    transit_gateway_attachment_id = data.aws_ec2_transit_gateway_peering_attachment.fix.id
    tags = {
        Name = "terraform-example"
        Side = "Acceptor"
    }
}

justinretzolk commented 2 years ago

Hey @pmcevoy 👋 Thank you for taking the time to raise this! I went through a reproduction for this, but I've not been able to replicate the behavior you experienced. I'll paste my example configuration below, in case there's anything that I've missed.

Edit: See next comment; this reproduction was flawed.

justinretzolk commented 2 years ago

Note: I'm going to re-work my configuration a bit, as I realized I had a flaw in my reproduction (I was not doing cross-account)

justinretzolk commented 2 years ago

Following up after further testing, I am still not able to reproduce the behavior described here. Below is my Terraform configuration. Note that with the way I've configured this reproduction, a targeted apply is needed for the aws_ec2_transit_gateway resources so that the aws_ec2_transit_gateway data sources will have something to pick up. This shouldn't impact the reproduction much, but I wanted to call it out regardless.

# Set the regions as a variable to make it easier to reference
variable "aws_first_region" {
  default = "us-east-1"
}

variable "aws_second_region" {
  default = "us-west-1"
}

# Profile variables so they're not visible from the configuration itself
variable "aws_first_profile" {}
variable "aws_second_profile" {}

# First accepts the Peering attachment.
provider "aws" {
  alias   = "first"
  region  = var.aws_first_region
  profile = var.aws_first_profile
}

# Second creates the Peering attachment.
provider "aws" {
  alias   = "second"
  region  = var.aws_second_region
  profile = var.aws_second_profile
}

data "aws_caller_identity" "first" {
  provider = aws.first
}

# For ease of reproduction, create the transit gateways in the same config
# Note: We won't actually reference these, since the issue report mentions using the data sources in place of direct reference
resource "aws_ec2_transit_gateway" "first" {
  provider = aws.first
  tags = {
    Name = "jretzolk-test-first-v1"
  }
}

resource "aws_ec2_transit_gateway" "second" {
  provider = aws.second
  tags = {
    Name = "jretzolk-test-second"
  }
}

# Use data sources to get the transit gateway IDs, to mirror the issue report
data "aws_ec2_transit_gateway" "first" {
  provider = aws.first

  filter {
    name   = "tag:Name"
    values = ["jretzolk-test-first-v1"]
  }
}

data "aws_ec2_transit_gateway" "second" {
  provider = aws.second

  filter {
    name   = "tag:Name"
    values = ["jretzolk-test-second"]
  }
}

# Create the Peering attachment in the second account...
resource "aws_ec2_transit_gateway_peering_attachment" "example" {
  provider                = aws.second
  peer_account_id         = data.aws_caller_identity.first.account_id
  peer_region             = var.aws_first_region
  peer_transit_gateway_id = data.aws_ec2_transit_gateway.first.id
  transit_gateway_id      = data.aws_ec2_transit_gateway.second.id
  tags = {
    Name = "jretzolk-test"
    Side = "Creator"
  }
}

# ...and accept it in the first account.
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example" {
  provider                      = aws.first
  transit_gateway_attachment_id = aws_ec2_transit_gateway_peering_attachment.example.id
  tags = {
    Name = "jretzolk-test"
    Side = "Acceptor"
  }
}

For a bit of verification, I've used terraform state show to inspect the peering attachment/accepter:

# attachment

$ terraform state show aws_ec2_transit_gateway_peering_attachment.example
# aws_ec2_transit_gateway_peering_attachment.example:
resource "aws_ec2_transit_gateway_peering_attachment" "example" {
    id                      = "tgw-attach-09ca59f88af333eee"
    peer_account_id         = "928<redacted>"
    peer_region             = "us-east-1"
    peer_transit_gateway_id = "tgw-0633a856c80366c39"
    tags                    = {
        "Name" = "jretzolk-test"
        "Side" = "Creator"
    }
    tags_all                = {
        "Name" = "jretzolk-test"
        "Side" = "Creator"
    }
    transit_gateway_id      = "tgw-0fb002a5c0cad45d9"
}


# accepter

$ terraform state show aws_ec2_transit_gateway_peering_attachment_accepter.example
# aws_ec2_transit_gateway_peering_attachment_accepter.example:
resource "aws_ec2_transit_gateway_peering_attachment_accepter" "example" {
    id                            = "tgw-attach-09ca59f88af333eee"
    peer_account_id               = "323<redacted>"
    peer_region                   = "us-west-1"
    peer_transit_gateway_id       = "tgw-0fb002a5c0cad45d9"
    tags                          = {
        "Name" = "jretzolk-test"
        "Side" = "Acceptor"
    }
    tags_all                      = {
        "Name" = "jretzolk-test"
        "Side" = "Acceptor"
    }
    transit_gateway_attachment_id = "tgw-attach-09ca59f88af333eee"
    transit_gateway_id            = "tgw-0633a856c80366c39"
}

pmcevoy commented 2 years ago

Thanks for taking the time to review and triage my issue. I can't explain what I was doing wrong. I copied the example verbatim - although I needed to use an {{assume_role}} snippet to setup alias=second and "data" to load existing TGWs

I could not get the attachment to become fully accepted using one aws_ec2_transit_gateway_peering_attachment_accepter resource: I needed to accept on both sides.

Nevertheless, my work around is robust, so I am happy to let this issue go if you would like to close it. At least this issue will document my observation for future-peoples :-)

heathsnow commented 2 years ago

I too need to use the 'workaround' as the example is incorrect. I wonder, @justinretzolk, ~if you're testing 2 regions in the same account instead of x-account~? Nevermind...I see your account ID prefixes differ. Hmmmm.

heathsnow commented 2 years ago

Okay @justinretzolk, I think your example above works specifically because you've added those data resources which are not present in the current example.

Your config:

  peer_transit_gateway_id = data.aws_ec2_transit_gateway.first.id
  transit_gateway_id      = data.aws_ec2_transit_gateway.second.id

Example config:

  peer_transit_gateway_id = aws_ec2_transit_gateway.first.id
  transit_gateway_id      = aws_ec2_transit_gateway.second.id

rvoitenko commented 2 years ago

I found that it's an AWS region-specific issue. At least for me. Code from example works perfectly for me in eu-west-1 region. But in eu-north-1 it's always the issue.

I see that in eu-north-1 peering attachment got different ID's in peer and requestor account, that's why aws_ec2_transit_gateway_peering_attachment_accepter can't find the ID given to it from a requestor.

This should be reported to AWS somehow.

crawforde commented 2 years ago

@rvoitenko I reported the inconsistent behavior recently to AWS. For now, I'm keeping an eye on the request for the terraform provider to write a data source for the more generic TGW attachment lookup endpoint, which offers more extensive filters and would enable some acceptable workarounds for this issue.

nzolot commented 1 year ago

I've faced same behaviour as @rvoitenko - transit gateway attachments ID are different for requester in 1st account and accepter in 2n account, thus it's not possible to use attribute 'id' of 'aws_ec2_transit_gateway_peering_attachment' resource as 'transit_gateway_attachment_id' parameter in 'aws_ec2_transit_gateway_peering_attachment_accepter'

The workaround that is mentioned by @pmcevoy is not usable - it will work only for the first run, only when peer attachment is in 'pendingAcceptance' state. Once the connection is accepted, the 'state' will be changed, as a result the data resource will fail (due to no results). This is not declarative/idempotent way and this not how terraform should work.

I believe there's some attribute which is missing in 'aws_ec2_transit_gateway_peering_attachment'- it should return ID of the attachment which has been created in another account, something like peer_id

Carlos4ndresh commented 1 year ago

I encountered this issue today. The code in https://github.com/hashicorp/terraform-provider-aws/blob/main/examples/transit-gateway-cross-account-peering-attachment/main.tf works just fine on accounts within the same organization; I know this because I've created one cross-account peering attachment with this code. However, I get this error when trying to do the same with accounts in different AWS organizations, upon which I can make changes.

Shouldn't it behave the same way if I have access to the two different AWS Orgs?

uo-thomas commented 1 year ago

I am seeing the same issue too and I think the change is when AWS made the change in Transit Gateway peering to include the same region. Originally the intention was to peer between two separate regions. Since both transit gateway attachments are in the same region and they are region in scope they need to have different IDs.

enzsadeddine commented 1 year ago

Hello, I had the same problem to peering between two transit gateway on the same account and the same region, the id of the attachment was not the one that allowed to accept, here is the solution that I made hoping that it will help you:

resource "aws_ec2_transit_gateway_peering_attachment" "peering_tgw" {
  peer_region             = var.region
  peer_transit_gateway_id = var.peer_transit_gateway.peer_transit_gateway_id
  transit_gateway_id      = aws_ec2_transit_gateway.transit.id

  tags = {
    Name = "peering-tgw-legacy"
  }
}

data "aws_ec2_transit_gateway_attachments" "attachments" {
  filter {
    name   = "state"
    values = ["pendingAcceptance"]
  }

  filter {
    name   = "resource-type"
    values = ["peering"]
  }

  depends_on = [aws_ec2_transit_gateway_peering_attachment.peering_tgw]
}

resource "aws_ec2_transit_gateway_peering_attachment_accepter" "peering_tgw_accepter" {
  transit_gateway_attachment_id = try(element([for attachment_id in data.aws_ec2_transit_gateway_attachments.attachments.ids : attachment_id if attachment_id != aws_ec2_transit_gateway_peering_attachment.peering_tgw.id], 0), "")

  tags = {
    Name = "peering-tgw-legacy-accepter"
  }

  depends_on = [aws_ec2_transit_gateway_peering_attachment.peering_tgw]

  lifecycle {
    ignore_changes = [transit_gateway_attachment_id]
  }
}

Tracy95 commented 1 year ago

I encountered same issue @crawforde. In same region different accounts. the peering attachment is different. If we use the data command to filter the peering attachment id in destination account. the existing filter criteria cannot uniquely identify the attachment id. The state is changeable @enzsadeddine. If we executed once, it is ok, but if we execute again. it would not find out the attachment id. In the console, There is Requester Id which can uniquely identify the attachment. Unfortunately, the Requester Id can not be used as filter.

mohamed-caylent commented 1 year ago

Until AWS/Hashicorp fixes the underlying issues, I ended up running local-exec in the aws_ec2_transit_gateway_peering_attachment that will assume a role in the other account, lookup the attachment and then accept it.

For me this issue occurs when I try to peer two transit gateways in the same region. The problem does not seem to happen if you try to peer two transit gateways in different regions.

robh007 commented 1 year ago

I encountered this issue today & it looks to be an issue when peering TGW across account(s). It doesn't happen when you peer between regions or within region providing the TGW(s) are in the same account, the IDs generated are the same. There's an additional attribute returned in the create response from the API AccepterTransitGatewayAttachmentId when it's across account.

{"TransitGatewayPeeringAttachment": {"TransitGatewayAttachmentId": "tgw-attach-0d563f5cc2146140b", "AccepterTransitGatewayAttachmentId": "tgw-attach-09efbd5b1be898b1b", "RequesterTgwInfo": {"TransitGatewayId": "tgw-004dc52edcf937e7f", "OwnerId": "123456789012", "Region": "eu-west-1"}, "AccepterTgwInfo": {"TransitGatewayId": "tgw-0180bfb4c7a8d076d", "OwnerId": "210987654321", "Region": "eu-west-1"}, "State": "initiatingRequest"}

On the describe this attribute isn't returned.

{"TransitGatewayAttachmentId": "tgw-attach-0d563d5cc2146140b", "RequesterTgwInfo": {"TransitGatewayId": "tgw-004de52edcf937e7f", "OwnerId": "123456789012", "Region": "eu-west-1"}, "AccepterTgwInfo": {"TransitGatewayId": "tgw-0180cfb4c7a8d076e", "OwnerId": "210987654321", "Region": "eu-west-1"}]}

I also can't seem to get the tests to run correctly.

make testacc TESTS=testAccTransitGatewayPeeringAttachmentAccepterConfig_differentAccount PKG=ec2   
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='testAccTransitGatewayPeeringAttachmentAccepterConfig_differentAccount'  -timeout 180m
testing: warning: no tests to run
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/ec2        3.954s [no tests to run]

flaviomoringa commented 11 months ago

I'm having the exact same same issue and I'm a stuck... createing the TGW peering accross accounts seems impossible right now using terraform because I cannot get the correct attach ID to pass to the accepter

This seems a big oversight :-(

github-actions[bot] commented 7 months ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

github-actions[bot] commented 7 months ago

This functionality has been released in v5.45.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions[bot] commented 6 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws