Using aws_region in module with module depends_on causes inconsistent plan on tags_all

[joey-coleman]

There appears to be an issue with inconsistent final plan error on tags_all when adding or overriding tags with values that are "known after apply".

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.3
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.52.0

Affected Resource(s)

• data.aws_region
• aws_ec2_transit_gateway_peering_attachment

Potentially others contained in a module; these are just the ones I was using

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

./main.tf

terraform {
  required_version = "~> 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

locals {
  tags = {
    Name = "Default Name"
  }
}

provider "aws" {
  alias   = "src"
  region  = "ca-central-1"
  default_tags {
    tags = local.tags
  }
}

provider "aws" {
  alias   = "dst"
  region  = "us-east-1"
  default_tags {
    tags = local.tags
  }
}

module "tgw_src" {
  source = "./tgw"
  providers = {
    aws = aws.src
  }
}

module "tgw_dst" {
  source = "./tgw"
  providers = {
    aws = aws.dst
  }
}

module "peering_ca-central-1_us-east-1" {
  ######
  # This depends_on causes an inconsistent plan
  depends_on = [module.tgw_src, module.tgw_dst]
  ######

  source     = "./peering"
  src_tgw_id = module.tgw_src.tgw_id
  dst_tgw_id = module.tgw_dst.tgw_id

  providers = {
    aws.src = aws.src
    aws.dst = aws.dst
  }
}

./tgw/tgw.tf

terraform {
  required_version = "~> 1.0"

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

data "aws_region" "current" {}

locals {
  asn_mapping = {
    ca-central-1 = 64512
    us-east-1    = 64513
  }
}

resource "aws_ec2_transit_gateway" "tgw" {
  amazon_side_asn                 = local.asn_mapping[data.aws_region.current.name]
  auto_accept_shared_attachments  = "disable"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"
  description                     = "TGW in ${data.aws_region.current.name}"
  dns_support                     = "enable"
}

resource "aws_ec2_transit_gateway_route_table" "peer" {
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id

  tags = {
    Name = "Peer TGW-RTB"
    Type = "Peer"
  }
}

output "tgw_id" {
  value = aws_ec2_transit_gateway.tgw.id
}

./peering/peering.tf

terraform {
  required_version = "~> 1.0"

  required_providers {
    aws = {
      source                = "hashicorp/aws"
      configuration_aliases = [aws.src, aws.dst]
    }
  }
}

variable "src_tgw_id" {
  type = string
}

variable "dst_tgw_id" {
  type = string
}

data "aws_region" "src" {
  provider = aws.src
}

data "aws_region" "dst" {
  provider = aws.dst
}

data "aws_ec2_transit_gateway" "src" {
  provider = aws.src
  id       = var.src_tgw_id
}

data "aws_ec2_transit_gateway" "dst" {
  provider = aws.dst
  id       = var.dst_tgw_id
}

data "aws_ec2_transit_gateway_route_table" "src" {
  provider = aws.src

  filter {
    name   = "transit-gateway-id"
    values = [data.aws_ec2_transit_gateway.src.id]
  }

  filter {
    name   = "tag:Type"
    values = ["Peer"]
  }
}

data "aws_ec2_transit_gateway_route_table" "dst" {
  provider = aws.dst

  filter {
    name   = "transit-gateway-id"
    values = [data.aws_ec2_transit_gateway.dst.id]
  }

  filter {
    name   = "tag:Type"
    values = ["Peer"]
  }
}

## This creates the peering link itself
resource "aws_ec2_transit_gateway_peering_attachment" "src" {
  provider                = aws.src
  peer_region             = data.aws_region.dst.name
  peer_account_id         = data.aws_ec2_transit_gateway.dst.owner_id
  peer_transit_gateway_id = data.aws_ec2_transit_gateway.dst.id
  transit_gateway_id      = data.aws_ec2_transit_gateway.src.id

  tags = {
    # ok with depends_on
    # Name = "Override Name - no interpolation"
    # ok with depends_on
    # Name = "Override Name - known interpolation ${path.root}"
    # error with depends_on
    Name = "Override Name - 'after apply' interpolation ${data.aws_region.dst.name}"
  }
}

Debug Output

Shell transcript from terraform apply of above script

Expected Behavior

The final plan should not have been inconsistent — if a resource's tags value is (known after apply) then terraform likely should not get upset that tags_all changed as a result.

Actual Behavior

An inconsistent final plan error as the tags changed after a data resource became available.

Steps to Reproduce

1. terraform apply on the files included above

Important Factoids

This is a cut-down example — I do (think I) need to use the modules in this way. In my full script, without depends_on, I get races between parts of of the tgw module finishing and the parts of the peering module that depend on those bits of the tgw module.

Without the depends_on in the module call sites, the data.aws_region data sources inside the peering module are fully determined in the plan, rather than being "known after apply". I don't think that has to be the case as the aws_region of a provide isn't going to change.

[j-sv]

I'm seeing the same issue with aws_route53_health_check and aws_ssm_parameter. This can triggered using localstack.

Start localstack with e.g. docker run --rm -it -e SERVICES=ssm -p 4566:4566 localstack/localstack:latest.

The minimal example I used was this:

./main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

provider "aws" {
  region                      = "us-east-1"
  access_key                  = "0"
  secret_key                  = "0"
  s3_force_path_style         = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    ssm = "http://localhost:4566"
  }

  default_tags {
    tags = {
      one = "two"
    }
  }
}

locals {
  name = "example"
}

resource "aws_ssm_parameter" "example" {
  name = local.name
  type = "String"
  value = "example"
}

module "m" {
  source = "./m"
  depends_on = [aws_ssm_parameter.example]

  name = local.name
}

./m/main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

data "aws_region" "current" {}

locals {
  name = "${data.aws_region.current.name}.${var.name}"
}

variable "name" {
  type = string
}

resource "aws_ssm_parameter" "example" {
  name  = local.name
  type  = "String"
  value = "example"

  tags = {
    name = local.name
  }
}

Worth to note is perhaps that if default_tags is removed on the provider the issue is not triggered.

[bassmanitram]

There are a large number of similar reports of this, the most active in terms of comments appears to be https://github.com/hashicorp/terraform-provider-aws/issues/19583

[ewbankkit]

Hey all 👋 Thank you very much for taking the time to raise this! This was addressed with #29747, which was included in version 5.0.0 of the provider. With that in mind, we'll close this issue. If you experience additional issues with the provider, please do open a new issue to let us know.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.