Unable to use network_interface_id within aws_route_table without incurring a diff everytime

[rene00]

Terraform Version

$ terraform -v
Terraform v0.10.1

Affected Resource(s)

• aws_route_table

Terraform Configuration Files

provider "aws" {
  region = "ap-southeast-2"
  alias  = "local"
}

resource "aws_vpc" "default" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "subnet1" {
  vpc_id            = "${aws_vpc.default.id}"
  availability_zone = "ap-southeast-2a"
  cidr_block        = "10.0.1.0/24"
}

resource "aws_subnet" "subnet2" {
  vpc_id            = "${aws_vpc.default.id}"
  availability_zone = "ap-southeast-2a"
  cidr_block        = "10.0.2.0/24"
}

resource "aws_instance" "server" {
  instance_type = "t2.nano"
  ami           = "ami-ae6259cd"
  subnet_id     = "${aws_subnet.subnet1.id}"
}

resource "aws_network_interface" "test" {
  subnet_id   = "${aws_subnet.subnet2.id}"
  private_ips = ["10.0.2.100"]

  attachment {
    instance     = "${aws_instance.server.id}"
    device_index = 1
  }
}

resource "aws_route_table" "rt" {
  vpc_id = "${aws_vpc.default.id}"

  route {
    cidr_block           = "10.1.1.1/32"
    network_interface_id = "${aws_network_interface.test.id}"
  }
}

Expected Behavior

Running terraform apply for the second time there should be no modifications.

Actual Behavior

Running terraform apply for the second time there are modifications.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform apply again.

Important Factoids

network_interface_id or instance_id can be set for routes within aws_route_table though when setting network_interface_id it appears AWS sends back instance_id AND network_interface_id which triggers a diff.

$ terraform apply
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.
$ terraform apply
...
aws_route_table.rt: Modifying... (ID: rtb-4b30d52c)
  route.1660468403.cidr_block:                "" => "10.1.1.1/32"
  route.1660468403.egress_only_gateway_id:    "" => ""
  route.1660468403.gateway_id:                "" => ""
  route.1660468403.instance_id:               "" => ""
  route.1660468403.ipv6_cidr_block:           "" => ""
  route.1660468403.nat_gateway_id:            "" => ""
  route.1660468403.network_interface_id:      "" => "eni-38498645"
  route.1660468403.vpc_peering_connection_id: "" => ""
  route.2141106289.cidr_block:                "10.1.1.1/32" => ""
  route.2141106289.egress_only_gateway_id:    "" => ""
  route.2141106289.gateway_id:                "" => ""
  route.2141106289.instance_id:               "i-06fbffdd7ceb7026f" => ""
  route.2141106289.ipv6_cidr_block:           "" => ""
  route.2141106289.nat_gateway_id:            "" => ""
  route.2141106289.network_interface_id:      "eni-38498645" => ""
  route.2141106289.vpc_peering_connection_id: "" => ""
aws_route_table.rt: Modifications complete (ID: rtb-4b30d52c)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

This looks like it was fixed for standalone routes with aws_route but not in-line routes within aws_route_table.

Work around for now is to replace network_interface_id with instance_id.

resource "aws_route_table" "rt" {
  vpc_id = "${aws_vpc.default.id}"

 route {
    cidr_block           = "10.1.1.1/32"
    instance_id = "${aws_instance.server.id}"
  }
}

References

• https://github.com/hashicorp/terraform/issues/4311

[rene00]

The workaround I had proposed won't always work if the instance has multiple interfaces. Received this error today:

1 error(s) occurred:

* aws_route_table.rtb-99dccefb: 1 error(s) occurred:

* aws_route_table.rtb-99dccefb: InvalidInstanceID: There are multiple interfaces attached to instance 'i-55324f0a'. Please specify an interface ID for the operation instead.
        status code: 400, request id: 7ad4f56a-03b6-4b9a-b5cf-2d0a187f028f

[catsby]

Hey @rene00 thank you for the issue and the proposed fix. We'll take a look or perhaps a community member will take a try at it

[psychoweb]

+1

Could it be possible to allow specifying both _instanceid and _network_interfaceid when defining aws_route_table objects? Currently this behaviour fails during configuration check and, while I see a demand of consistency between the two IDs, AWS itself could give an error if the specified network interface is not attached to the relative instance.

[nevir]

Work around for now is to replace network_interface_id with instance_id.

This occurs for me when providing instance_id (and not network_interface_id), too :(

[fabiodbr]

An aws_route_table using an instance as VPN occurred diff every time for me too using an network_interface_id in the route. The workaround was to create an aws_route separately.

resource "aws_route_table" "private" { vpc_id = "${aws_vpc.this.id}" }

resource "aws_route" "natgw" { route_table_id = "${aws_route_table.private.id}" nat_gateway_id = "${aws_nat_gateway.nat_gw.id}" destination_cidr_block = "0.0.0.0/0" }

resource "aws_route" "vpn" { route_table_id = "${aws_route_table.private.id}" network_interface_id = "${aws_instance.vpn_instance.primary_network_interface_id}" destination_cidr_block = "10.10.0.0/22" }

[klatiss]

I'm still seeing this with Terraform 0.11.14 and the latest AWS provider (2.11.0). My current workaround is to ignore route changes in a lifycycle block after initial creation. I can do this due to a somewhat static environment but it probably wont work for everyone.

[orgito]

Terraform v0.12.3 and aws provider v2.17.0 and the problem persists.

[ckabalan]

Still seeing this with Terraform v0.12.20 and the AWS Provider v2.48.0.

[ewbankkit]

Relates to the set of issues with aws_route addressed in https://github.com/hashicorp/terraform-provider-aws/pull/16930. This can be solved by a breaking change (removing instance_id attribute): https://github.com/hashicorp/terraform-provider-aws/issues/14197.

[sylr]

Since I've encountered this issue myself (which I deem critical in my setup) I've taken it upon myself to implement #14197, build and publish it on the terraform registry.

So for those in need of a fix for this issue you can use sylr/aws. The source code can be found at https://github.com/sylr/terraform-provider-aws/tree/v3.71.0-sylr.1.

[tullydwyer]

Still getting a perpetual diff with Terraform 1.2.1 and AWS Provider 4.24.0

[unixtastic]

Still getting this on Terraform 1.2.8 and AWS Provider 4.11.0

Can we get rid of the warning saying to use network_interface_id instead of instance_id? network_interface_id doesn't seem to be a sensible option right now.

[jtele2]

Still get this error on Terraform 1.2.9 and AWS Provider 4.31.0.

Agree with @unixtastic - the warning is annoying, but using instance_id is the only viable option to avoid constant diffs.

[paulopatto]

Still get the error on:

terraform --version 
Terraform v1.3.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.49.0

My code basicaly:

resource "aws_eip" "nat_gateway_ip" {
  vpc = true

  tags = {
    Name = "Public IP to Privnet"
  }
}

resource "aws_nat_gateway" "nat_gateway" {
  allocation_id = aws_eip.nat_gateway_ip.id
  subnet_id     = aws_subnet.dmz.id

  tags          = {
    Name = "Private network nat gateway"
    Description = "NAT gateway 4 private instances"
  }
}

resource "aws_route_table" "internal" {
  vpc_id = aws_vpc.lab.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_nat_gateway.nat_gateway.id
  }

  tags = {
    Name = "Internal route table"
  }
}

resource "aws_route_table_association" "private_subnets" {
  # List of subnets created w/ count
  for_each = { for k, subnet in aws_subnet.private : k => subnet }

  subnet_id      = each.value.id
  route_table_id = aws_route_table.internal.id
}

And every plan or apply:

Actual Behaviour

Every time I exec plan || apply, there are modifications.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_route_table.internal will be updated in-place
  ~ resource "aws_route_table" "internal" {
        id               = "rtb-085553f49104ea7f9"
      ~ route            = [
          - {
              - carrier_gateway_id         = ""
              - cidr_block                 = "0.0.0.0/0"
              - core_network_arn           = ""
              - destination_prefix_list_id = ""
              - egress_only_gateway_id     = ""
              - gateway_id                 = ""
              - instance_id                = ""
              - ipv6_cidr_block            = ""
              - local_gateway_id           = ""
              - nat_gateway_id             = "nat-0f5a04ec428f6e340"
              - network_interface_id       = ""
              - transit_gateway_id         = ""
              - vpc_endpoint_id            = ""
              - vpc_peering_connection_id  = ""
            },
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = "nat-0f5a04ec428f6e340"
              + instance_id                = ""
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = ""
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
        tags             = {
            "Name" = "Tabela de rotas interna"
        }
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Expected behaviour

No changes

[jbohmann]

@paulopatto Looks like you might be seeing that behavior for a different reason: gateway_id vs. nat_gateway_id. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/route_table

NOTE on gateway_id and nat_gateway_id: The AWS API is very forgiving with these two attributes and the aws_route_table resource can be created with a NAT ID specified as a Gateway ID attribute. This will lead to a permanent diff between your configuration and statefile, as the API returns the correct parameters in the returned route table. If you're experiencing constant diffs in your aws_route_table resources, the first thing to check is whether or not you're specifying a NAT ID instead of a Gateway ID, or vice-versa.

In the route config, try changing gateway_id to nat_gateway_id:

route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway.id
}

[MonkadelicD]

$> terraform --version Terraform v1.4.5 on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.46.0

This issue is very frustrating when route tables get extensive. Finding what's actually changing in the output of 'terraform plan' is very time consuming. How has this not been resolved after being a bug since 2017? I'll have to put a lot of work into converting route blocks within aws_route_table resource blocks to individual aws_route resource blocks as mentioned by @fabiodbr

Some mention of this issue should be included in the AWS module documentation page for "Resource: aws_route_table", instead all we have is a recommendation to use network_interface_id instead of instance_id.

[unixtastic]

Is it possible to add a warning to the aws_route_table documentation recommending against the use of in-line routes?

This should not be left as an undocumented trap for future users.

[jar-b]

Closed by #30804, merged to main via #31392

[github-actions[bot]]

This functionality has been released in v5.0.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.