multi-account example does not work when using multiple AWS accounts

mattwilder commented 11 months ago

Description

I am in the process of rolling out a transit gateway network with attached VPCs in multiple AWS accounts. I followed the example provided in the multi-account directory, but the TGW account side will fail with the following error:

│ Error: creating EC2 Transit Gateway VPC Attachment: InvalidSubnetID.NotFound: The subnet ID <peer account subnet id> does not exist

NOTE: subnet id replaced by me for clarity

The subnet in question definitely does exist, but the provider used by the resources in the TGW account can not see it because it lives in the peer account.

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]: 2.10.0
Terraform version: Terraform v1.3.9
Provider version(s):
```
Terraform v1.3.9
on linux_amd64
```
provider registry.terraform.io/hashicorp/aws v5.29.0

Reproduction Code [Required]

Deploy the multi-account example with two different accounts by configuring the providers to use IAM roles in each account that have full administrative access to each account.

Our AWS infrastructure uses AWS Organizations so the administrative IAM role is named OrganizationAccountAccessRole, but any IAM role with sufficient access will do.

For example:

provider "aws" {
  region = local.region
  assume_role {
    role_arn = "arn:aws:iam::<account-1>:role/OrganizationAccountAccessRole"
  }
}

# This provider is required for attachment only installation in another AWS Account
provider "aws" {
  region = local.region
  alias  = "peer"
  assume_role {
    role_arn = "arn:aws:iam::<account-2>:role/OrganizationAccountAccessRole"
  }
}

Steps to reproduce the behavior: Not using workspaces and cache was cleared

Setup provider as shown above
run terraform init and terraform apply

Expected behavior

The multi account example should work

Actual behavior

Fails with

│ Error: creating EC2 Transit Gateway VPC Attachment: InvalidSubnetID.NotFound: The subnet ID <peer account subnet id> does not exist

NOTE: I replaced the actual subnet id with <peer account subnet id> to make the error message more understandable

lpsm-dev commented 10 months ago

Same error

sts-manuel commented 10 months ago

i have the same issue.

sarasensible commented 9 months ago

I had to add my peer vpc attachment separately and create the routes separately as well for this to work.

main.tf


############ Data Sources ###########

data "aws_vpcs" "prod_vpc" {
  tags = {
    Terraform = "true"
  }
}

data "aws_vpc" "prod_info" {
  id    = data.aws_vpcs.prod_vpc.ids[0]
}

data "aws_vpcs" "dev_vpc" {
  tags = {
    Terraform = "true"
  }
  provider = aws.peer
}

data "aws_vpc" "dev_info" {
  id    = data.aws_vpcs.dev_vpc.ids[0]
  provider = aws.peer
}

data "aws_subnets" "prod_private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.prod_info.id]
  }
  tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

data "aws_subnet" "prod_private_info" {
  for_each = toset(data.aws_subnets.prod_private.ids)
  id = each.value
}

data "aws_route_tables" "prod_route_tables" {
  vpc_id = data.aws_vpc.prod_info.id
}

data "aws_subnets" "dev_private" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.dev_info.id]
  }
  tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
  provider = aws.peer
}

data "aws_subnet" "dev_private_info" {
  for_each = toset(data.aws_subnets.dev_private.ids)
  id = each.value
  provider = aws.peer
}

data "aws_route_tables" "dev_route_tables" {
  vpc_id = data.aws_vpc.dev_info.id
  provider = aws.peer
}

data "aws_ec2_transit_gateway_route_tables" "tgw_rtb" {
  depends_on = [module.tgw]
  filter {
    name   = "transit-gateway-id"
    values = [module.tgw.ec2_transit_gateway_id]
  }
  filter {
    name   = "default-association-route-table"
    values = [true]
  }
}

############ Transit Gateway ###########

module "tgw" {
  source  = "terraform-aws-modules/transit-gateway/aws"
  version = "~> 2.12.1"

  name            = var.tgw_name
  description     = var.tgw_description
  amazon_side_asn = 64532

  enable_dns_support = true

  transit_gateway_cidr_blocks = [cidrsubnet(data.aws_vpc.prod_info.cidr_block, 4, 6)]

  enable_auto_accept_shared_attachments = true

  enable_multicast_support = false

  create_tgw_routes = false

  vpc_attachments = {
    prod_vpc = {
      vpc_id     = data.aws_vpc.prod_info.id
      subnet_ids = local.prod_private_subnets
      dns_support  = true
      ipv6_support = false

      transit_gateway_default_route_table_association = true
      transit_gateway_default_route_table_propagation = true

      tags = var.tags
    },
  }

  ram_name = var.ram_name
  ram_allow_external_principals = true
  ram_principals                = [var.dev_account]

  tags = var.tags
}

resource "aws_ec2_transit_gateway_vpc_attachment" "dev" {
  depends_on = [module.tgw]
  provider = aws.peer
  transit_gateway_id = module.tgw.ec2_transit_gateway_id
  vpc_id             = data.aws_vpc.dev_info.id
  subnet_ids         = local.dev_private_subnets

  dns_support                                     = "enable"
  ipv6_support                                    = "disable"
  appliance_mode_support                          = "disable"
  transit_gateway_default_route_table_association = true
  transit_gateway_default_route_table_propagation = true

  tags = var.tags
}

resource "aws_ec2_transit_gateway_route" "prod" {
  depends_on = [module.tgw]
  for_each = local.tgw_routes
  destination_cidr_block = each.value.destination_cidr_block
  blackhole              = try(each.value.blackhole, false)

  transit_gateway_route_table_id = data.aws_ec2_transit_gateway_route_tables.tgw_rtb.ids[0]
  transit_gateway_attachment_id  = each.value.attach_id
}

resource "aws_route" "dest_prod" {
  for_each = toset(data.aws_route_tables.prod_route_tables.ids)

  route_table_id              = each.value
  destination_cidr_block      = var.dest_cloud_subnet
  destination_ipv6_cidr_block = null
  transit_gateway_id          = module.tgw.ec2_transit_gateway_id
}

resource "aws_route" "dest_dev" {
  depends_on = [aws_ec2_transit_gateway_vpc_attachment.dev]
  for_each = toset(data.aws_route_tables.dev_route_tables.ids)
  provider = aws.peer

  route_table_id              = each.value
  destination_cidr_block      = var.dest_cloud_subnet
  destination_ipv6_cidr_block = null
  transit_gateway_id          = module.tgw.ec2_transit_gateway_id
}

locals.tf

locals {
  prod_subnet_az_to_id   = zipmap(values(data.aws_subnet.prod_private_info).*.availability_zone, values(data.aws_subnet.prod_private_info).*.id)
  prod_sorted_subnet_azs = sort(values(data.aws_subnet.prod_private_info).*.availability_zone)

  prod_private_subnets = flatten([
    for az, id in local.prod_subnet_az_to_id : [
      for sorted_az in local.prod_sorted_subnet_azs :
            id if az == sorted_az
      ]
    ])

  dev_subnet_az_to_id   = zipmap(values(data.aws_subnet.dev_private_info).*.availability_zone, values(data.aws_subnet.dev_private_info).*.id)
  dev_sorted_subnet_azs = sort(values(data.aws_subnet.dev_private_info).*.availability_zone)

  dev_private_subnets = flatten([
    for az, id in local.dev_subnet_az_to_id : [
      for sorted_az in local.dev_sorted_subnet_azs :
            id if az == sorted_az
      ]
    ])

    tgw_routes_map = [
      {
          destination_cidr_block = data.aws_vpc.prod_info.cidr_block
          attach_id = module.tgw.ec2_transit_gateway_vpc_attachment_ids[0]
        },
        {
          destination_cidr_block = data.aws_vpc.dev_info.cidr_block
          attach_id = aws_ec2_transit_gateway_vpc_attachment.dev.id
        }
      ]
  tgw_routes = tomap({for r in local.tgw_routes_map:  r.destination_cidr_block => r})

}

variables.tf

variable env {
  description = "Environment"
  default = null
}

variable aws_account {
  description = "Account to provision in"
  default = null
}

variable dev_account {
  description = "Dev/Peer Account to provision in"
  default = null
}

variable region {
  description = "Region of the cluster"
  default = "us-east-2"
}

variable "dest_cloud_subnet" {
  description = "Dest Subnet in CIDR form"
  type = string
  default = null
}

variable "tags" {
  description = "Resource tags"
  type = any
  default = null
}

variable "tgw_name" {
  description = "TGW Name"
  type = string
  default = null
}

variable "tgw_description" {
  description = "TGW Description"
  type = string
  default = null
}

variable "ram_name" {
  description = "RAM Name"
  type = string
  default = null
}

Hope it helps!

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

mattwilder commented 8 months ago

ping

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

mattwilder commented 7 months ago

ping

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

github-actions[bot] commented 5 months ago

This issue was automatically closed because of stale in 10 days

github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

terraform-aws-modules / terraform-aws-transit-gateway