Resource: aws_ec2_transit_gateway lacks the ability to set the default association and propagation route tables

[boco372]

The AWS API has the ability to set specific tgw-rtbs for association and propagation - I can't find this ability in the terraform aws provider (I'm using version 3.26).

This page describes how to set these in the CLI: https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-transit-gateway.html

We're trying to create a setup like this example here: https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-appliance-scenario.html

There are two different TGW route tables required - one for association, one for propagation. Unless I missed something (always likely) there is no way to do this in terraform code.

This can't be done as part of the TGW creation as it depends on knowing the route table IDs, and we can't create route tables till we have a TGW to put them in, so this 'modify' action would appear to need to be done after initial creation.

Thanks.

[ewbankkit]

@boco372 Thanks for raising this issue. The functionality you are looking for should be available via the aws_ec2_transit_gateway_route_table_association and aws_ec2_transit_gateway_route_table_propagation resources.

[boco372]

Hi there, Thanks for looking at this issue so promptly. The resources you suggest probably work for some use cases. I should have given more detail in the case. We have multiple accounts (under the same Organisation). The shared account creates the TGW and makes it available via RAM share. Other accounts can attach to this TGW via this share. These 'other' accounts can't see what route tables are available in the TGW share, and I'm unable to add the route tables to the RAM share. The only way to get my 'other' VPC's to connect to the correct association and propagation route tables is for them to use the defaults provided by the TGW. And thats my issue, I'm unable to set the correct route tables to be the association share and propagation share for this particular TGW. I can use the resources you suggest when both VPCS are in the same account, but not when they are in different accounts.

Apologies for not making that clear.

Thanks.

[desweil]

same here. I need to change these two settings after creation of TGW and TGW Routing Tables. It works via modify-transit-gateway on AWS CLI (AssociationDefaultRouteTableId and PropagationDefaultRouteTableId). I do not want to rely on local-exec however.

[desweil]

@boco372 Thanks for raising this issue. The functionality you are looking for should be available via the aws_ec2_transit_gateway_route_table_association and aws_ec2_transit_gateway_route_table_propagation resources.

This works for manually assigning attachments to Routing Tables. But we want to change the defaults so that we do NOT have to do this manually. This is needed for complex scenarios where TGW is shared to many accounts and VPCs.

[elabbarw]

I hope you don't mind me bumping this as well.

We need to set the attributes of the association and propagation route table ids on the transit gateway. Currently, enabling:

default_route_table_association default_route_table_propagation

would just end up creating an extra route table outside of the TF deployment and linking it to the TGW, but we want to associate the transit gateway to a default route table that we defined in the same state.

It seems the attributes of:

association_default_route_table_id propagation_default_route_table_id

Are not being tracked at all. I can deploy all the resources and then go into the GUI to change the default route tables manually without any complaint. But of course with everything else nicely linked together and dynamic it is just messy. They show up under unchanged attributes as per this example from Terraform Enterprise even though I changed them:

[ahewittbell]

I Agree, the ability to set the default route table and association table would be a nice addition. This is also available via the AWS CLI.

The work around I currently have in place is:

resource "null_resource" "main" {
  provisioner "local-exec" {
    command = "aws ec2 modify-transit-gateway --transit-gateway-id ${TGW_ID} --options AssociationDefaultRouteTableId=${TGW_RT_ID},PropagationDefaultRouteTableId=${TGW_RT_ID}"
  }
}

[CarterSheehan]

The above workaround works as expected with one issue: On destroy the TGW route table throws the error: Error: error deleting EC2 Transit Gateway Route Table: IncorrectState: <ROUTE_TABLE_ID REMOVED> is set as default propagation route table for <TGW_ID REMOVED>

To get around this, I tweaked the null_resource with some tricks from this thread

In the end, the null_resource looks like this:

resource "null_resource" "modify_tgw_to_enable_default_route_propagation" {
  triggers = {
    trigger            = jsonencode(aws_ec2_transit_gateway.hub)
    aws_region         = var.aws_region
    transit_gateway_id = aws_ec2_transit_gateway.hub.id
  }

  lifecycle {
    ignore_changes = [triggers["aws_region"], triggers["transit_gateway_id"]]
  }

  provisioner "local-exec" {
    command = "aws ec2 modify-transit-gateway --transit-gateway-id ${aws_ec2_transit_gateway.hub.id} --options DefaultRouteTablePropagation=enable,PropagationDefaultRouteTableId=${aws_ec2_transit_gateway_route_table.return.id} --region ${var.aws_region}"
  }

  provisioner "local-exec" {
    when    = destroy
    command = "aws ec2 modify-transit-gateway --transit-gateway-id ${self.triggers.transit_gateway_id} --options DefaultRouteTablePropagation=disable --region ${self.triggers.aws_region}"
  }
}

Since a destroy provisioner can only reference 'self', 'count', or 'each' so, I needed to work around this limitation. In order to get the transit gateway id and aws region passed into the command, I set a trigger and a lifecycle rule. I added the trigger because triggers can be accessed from the 'self' reference and I added the lifecycle rule so the value is set once at creation time and any changes are ignored so we don't replace this null_resource unless the real trigger changes.

I could not add this destroy provisioner to the route_table resource directly because again, we cannot access external data and since the aws_region is not available within the route_table resource, I was forced to move it elsewhere.

The only caveat to this is that the new triggers will not be added on a new apply for some reason. I had to destroy the null_resource before applying this change to get the triggers so the destroy provisioner would work. I did this by running a destroy in -target mode with the current master branch, then I applied this change in target mode to a new null_resource then ran an entire module destroy.

[SizZiKe]

bumping this / +1, this seems like an oversight...these attributes should be able to be managed via aws_ec2_transit_gateway @ewbankkit

[onitake]

One important thing to keep in mind: Since the CreateTransitGateway API doesn't have parameters to specify route table IDs, it will probably always create a new route table.

If assignment parameters are added to the aws_ec2_transit_gateway resource, they would require a separate ModifyTransitGateway call, and the automatically created route table would remain.

I'm not sure if creating the default route table is preventable (maybe with DefaultRouteTableAssociation=false and DefaultRouteTablePropagation=false ?), but if it isn't, an additional DeleteTransitGatewayRouteTable may be necessary.

[natewarr]

@CarterSheehan Great idea. I am using cross-account roles, so here's a way to do this in that circumstance. Hack on top of hack.

locals {
  tf_assumed_role = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${split("/", data.aws_caller_identity.current.arn)[1]}"
}

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

resource "null_resource" "change_default_propagation_route_table" {
  triggers = {
    trigger                = jsonencode(aws_ec2_transit_gateway.tgw)
    aws_region             = data.aws_region.current.name
    tf_caller_identity_arn = data.aws_caller_identity.current.arn
    tf_assumed_role        = local.tf_assumed_role
    transit_gateway_id     = aws_ec2_transit_gateway.tgw.id
  }

  lifecycle {
    ignore_changes = [triggers["aws_region"], triggers["tf_caller_identity"], triggers["tf_assumed_role"], triggers["transit_gateway_id"] ]
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      AWS_DEFAULT_REGION = self.triggers.aws_region
    }
    command = <<EOF
set -e

shell_id="$(aws sts get-caller-identity --query Arn --output text)"

if [ "${self.triggers.tf_caller_identity_arn}" != "$shell_id" ]; then

  CREDENTIALS=(`aws sts assume-role \
    --role-arn ${local.tf_assumed_role} \
    --role-session-name "terraform_nullresource_tgw" \
    --query "[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]" \
    --output text`)

  unset AWS_PROFILE
  export AWS_DEFAULT_REGION=us-east-1
  export AWS_ACCESS_KEY_ID="$${CREDENTIALS[0]}"
  export AWS_SECRET_ACCESS_KEY="$${CREDENTIALS[1]}"
  export AWS_SESSION_TOKEN="$${CREDENTIALS[2]}"
fi

aws ec2 modify-transit-gateway --transit-gateway-id ${aws_ec2_transit_gateway.tgw.id} --options \
  DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable,AssociationDefaultRouteTableId=${aws_ec2_transit_gateway_route_table.spoke.id},PropagationDefaultRouteTableId=${aws_ec2_transit_gateway_route_table.security.id}
EOF
  }

  provisioner "local-exec" {
    when        = destroy
    interpreter = ["/bin/bash", "-c"]
    environment = {
      AWS_DEFAULT_REGION = self.triggers.aws_region
    }
    command     = <<EOF2
set -e

shell_id="$(aws sts get-caller-identity --query Arn --output text)"

if [ "${self.triggers.tf_caller_identity_arn}" != "$shell_id" ]; then

  CREDENTIALS=(`aws sts assume-role \
    --role-arn ${self.triggers.tf_assumed_role} \
    --role-session-name "terraform_nullresource_tgw" \
    --query "[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]" \
    --output text`)

  unset AWS_PROFILE
  export AWS_DEFAULT_REGION=us-east-1
  export AWS_ACCESS_KEY_ID="$${CREDENTIALS[0]}"
  export AWS_SECRET_ACCESS_KEY="$${CREDENTIALS[1]}"
  export AWS_SESSION_TOKEN="$${CREDENTIALS[2]}"
fi

aws ec2 modify-transit-gateway --transit-gateway-id ${self.triggers.transit_gateway_id} --options DefaultRouteTableAssociation=disable,DefaultRouteTablePropagation=disable
EOF2

  }

}

[fernandosalomao]

@CarterSheehan Great idea. I am using cross-account roles, so here's a way to do this in that circumstance. Hack on top of hack.

locals {
  tf_assumed_role = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${split("/", data.aws_caller_identity.current.arn)[1]}"
}

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

resource "null_resource" "change_default_propagation_route_table" {
  triggers = {
    trigger                = jsonencode(aws_ec2_transit_gateway.tgw)
    aws_region             = data.aws_region.current.name
    tf_caller_identity_arn = data.aws_caller_identity.current.arn
    tf_assumed_role        = local.tf_assumed_role
    transit_gateway_id     = aws_ec2_transit_gateway.tgw.id
  }

  lifecycle {
    ignore_changes = [triggers["aws_region"], triggers["tf_caller_identity"], triggers["tf_assumed_role"], triggers["transit_gateway_id"] ]
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      AWS_DEFAULT_REGION = self.triggers.aws_region
    }
    command = <<EOF
set -e

shell_id="$(aws sts get-caller-identity --query Arn --output text)"

if [ "${self.triggers.tf_caller_identity_arn}" != "$shell_id" ]; then

  CREDENTIALS=(`aws sts assume-role \
    --role-arn ${local.tf_assumed_role} \
    --role-session-name "terraform_nullresource_tgw" \
    --query "[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]" \
    --output text`)

  unset AWS_PROFILE
  export AWS_DEFAULT_REGION=us-east-1
  export AWS_ACCESS_KEY_ID="$${CREDENTIALS[0]}"
  export AWS_SECRET_ACCESS_KEY="$${CREDENTIALS[1]}"
  export AWS_SESSION_TOKEN="$${CREDENTIALS[2]}"
fi

aws ec2 modify-transit-gateway --transit-gateway-id ${aws_ec2_transit_gateway.tgw.id} --options \
  DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable,AssociationDefaultRouteTableId=${aws_ec2_transit_gateway_route_table.spoke.id},PropagationDefaultRouteTableId=${aws_ec2_transit_gateway_route_table.security.id}
EOF
  }

  provisioner "local-exec" {
    when        = destroy
    interpreter = ["/bin/bash", "-c"]
    environment = {
      AWS_DEFAULT_REGION = self.triggers.aws_region
    }
    command     = <<EOF2
set -e

shell_id="$(aws sts get-caller-identity --query Arn --output text)"

if [ "${self.triggers.tf_caller_identity_arn}" != "$shell_id" ]; then

  CREDENTIALS=(`aws sts assume-role \
    --role-arn ${self.triggers.tf_assumed_role} \
    --role-session-name "terraform_nullresource_tgw" \
    --query "[Credentials.AccessKeyId,Credentials.SecretAccessKey,Credentials.SessionToken]" \
    --output text`)

  unset AWS_PROFILE
  export AWS_DEFAULT_REGION=us-east-1
  export AWS_ACCESS_KEY_ID="$${CREDENTIALS[0]}"
  export AWS_SECRET_ACCESS_KEY="$${CREDENTIALS[1]}"
  export AWS_SESSION_TOKEN="$${CREDENTIALS[2]}"
fi

aws ec2 modify-transit-gateway --transit-gateway-id ${self.triggers.transit_gateway_id} --options DefaultRouteTableAssociation=disable,DefaultRouteTablePropagation=disable
EOF2

  }

}

@natewarr Thanks, I also use crossaccount roles and this is the piece that was missing :)

I've modified a bit as I would prefer using aws cli configure to manage assume role.

resource "null_resource" "modify_tgw_to_enable_default_route_association_propagation" {
  triggers = {
    trigger                            = jsonencode(aws_ec2_transit_gateway.main)
    tf_assumed_role                    = local.tf_assume_role
    transit_gateway_id                 = aws_ec2_transit_gateway.main.id
    association_default_route_table_id = local.tgw_association_default_route_table_id
    propagation_default_route_table_id = local.tgw_propagation_default_route_table_id
  }

  lifecycle {
    ignore_changes = [triggers["tf_assumed_role"], triggers["transit_gateway_id"], triggers["association_default_route_table_id"], triggers["propagation_default_route_table_id"]]
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = <<EOF
aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
aws configure set --profile terraform region $AWS_REGION
aws configure set --profile terraform source_profile default
aws configure set --profile terraform role_arn ${self.triggers.tf_assumed_role}
aws configure set --profile terraform role_session_name "terraform_nullresource_tgw_modify"

aws ec2 --profile terraform modify-transit-gateway --transit-gateway-id ${self.triggers.transit_gateway_id} --options \
  DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable,AssociationDefaultRouteTableId=${self.triggers.association_default_route_table_id},PropagationDefaultRouteTableId=${self.triggers.propagation_default_route_table_id}
EOF
  }

  provisioner "local-exec" {
    when        = destroy
    interpreter = ["/bin/bash", "-c"]
    command     = <<EOF2
aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
aws configure set --profile terraform region $AWS_REGION
aws configure set --profile terraform source_profile default
aws configure set --profile terraform role_arn ${self.triggers.tf_assumed_role}
aws configure set role_session_name "terraform_nullresource_tgw_destroy"

aws ec2 --profile terraform modify-transit-gateway --transit-gateway-id ${self.triggers.transit_gateway_id} --options DefaultRouteTableAssociation=disable,DefaultRouteTablePropagation=disable
EOF2

  }

}

[blacksheep--]

bumping this, as the current situation really complicates things

[fumantsu]

Any update/plan on this? We also have some use case and using local-exec for the aws cli is I would say... counter-productive.

[SlavaSubotskiy]

There is one more workaround using import block, to use the default route table:

resource "aws_ec2_transit_gateway" "example" {
  description = "example"
}

import {
  to = aws_ec2_transit_gateway_route_table.example
  id = aws_ec2_transit_gateway.example.association_default_route_table_id
}

resource "aws_ec2_transit_gateway_route_table" "example" {
  transit_gateway_id = aws_ec2_transit_gateway.example.id
  tags = {
    Name = "example"
  }
}

Unfortunately, this code cannot be applied in one-shut. We should apply TGW first, e.g.: terraform apply -target aws_ec2_transit_gateway.example.