Discussion: constructing `vpc_az_maps` deterministically

[oponomarov-tu]

Long time no see!

Since recently I've started incorporating AWS-maintained Terraform module for provisioning VPCs mainly due to its better support of AWS IPAM. Trying to construct the vpc_az_maps from VPC module's outputs and pass it down to AlterNAT module, I've discovered that Terraform can't plan the changes. Note: using targeted applies and applying VPC module first & AlterNAT module afterwards works w/o issues.

I'm looking forward to any ideas how this could be solved in painless manner. Would love to contribute the solution into examples if we manage to find one.

💡 CloudPosse had written a nice article about values that can't be determined until apply, leaving it here for those who might come across.

---

The following Terraform (slightly simplified for demonstration purposes):

module "vpc_from_cidr" {
  count = 1

  source  = "aws-ia/vpc/aws"
  version = "~> 4.4.2"

  name     = "example-vpc"
  az_count = 3

  cidr_block = "172.16.44.0/22"

  subnets = {
    private = { netmask = 26 }
    public  = { netmask = 26 }
  }
}

locals {
  private_subnet_attributes_by_az      = module.vpc_from_cidr[0].private_subnet_attributes_by_az
  private_route_table_attributes_by_az = module.vpc_from_cidr[0].rt_attributes_by_type_by_az.private
  public_subnet_attributes_by_az       = module.vpc_from_cidr[0].public_subnet_attributes_by_az

  vpc_config_for_alternat = [
    for name, attributes in local.private_subnet_attributes_by_az : {
      az                 = attributes.availability_zone
      private_subnet_ids = [attributes.id]
      public_subnet_id   = local.public_subnet_attributes_by_az[attributes.availability_zone].id
      route_table_ids = [
        local.private_route_table_attributes_by_az[name].id
      ]
    }
  ]
}

module "alternat_instances" {
  source  = "chime/alternat/aws"
  version = "0.6.0"

  nat_instance_type   = "t4g.medium"
  lambda_package_type = "Zip"

  vpc_id      = module.vpc_from_cidr[0].vpc_attributes.id
  vpc_az_maps = local.vpc_config_for_alternat
}

.. results in:

│ Error: Invalid for_each argument
│
│   on .terraform/modules/cell.alternat_instances/lambda.tf line 129, in resource "aws_lambda_function" "alternat_connectivity_tester":
│  129:   for_each = { for obj in var.vpc_az_maps : obj.az => obj }
│     ├────────────────
│     │ var.vpc_az_maps is a list of object, known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵
╷
│ Error: Invalid count argument
│
│   on .terraform/modules/cell.alternat_instances/main.tf line 55, in resource "aws_eip" "nat_instance_eips":
│   55:   count = local.reuse_nat_instance_eips ? 0 : length(var.vpc_az_maps)
│
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.
╵
╷
│ Error: Invalid for_each argument
│
│   on .terraform/modules/cell.alternat_instances/main.tf line 425, in resource "aws_eip" "nat_gateway_eips":
│  425:   for_each = {
│  426:     for obj in var.vpc_az_maps
│  427:     : obj.az => obj.public_subnet_id
│  428:     if var.create_nat_gateways
│  429:   }
│     ├────────────────
│     │ var.create_nat_gateways is true
│     │ var.vpc_az_maps is a list of object, known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

[bwhaley]

The error is because the key in some of the for_each blocks is the az of the vpc_az_map, and that value is computed with data.aws_subnet.subnet[index].availability_zone when constructing the map. If you are able to hardcode the AZ in the map it should work. e.g.

locals {
  azs = ["us-east-1a", "us-east-1b"]
  vpc_az_maps = [
    for index, az in local.azs
    : {
      az                 = az
      route_table_ids    = [module.vpc.private_route_table_ids[index]]
      public_subnet_id   = module.vpc.public_subnets[index]
      private_subnet_ids = [module.vpc.private_subnets[index]]
    }
  ]
}

[oponomarov-tu]

Thank you, @bwhaley! I've got it working eventually with the following:

main.tf

```hcl variable "az_count" { description = "Number of AZs to use." type = number default = 3 } data "aws_availability_zones" "current" { filter { name = "opt-in-status" values = ["opt-in-not-required"] } } locals { # Searches region for # of AZs to use and takes a slice based on count. # Assume slice is sorted a-z. azs = slice(data.aws_availability_zones.current.names, 0, var.az_count) vpc_name = "example-vpc" vpc_cidr = "172.16.44.0/22" subnets = { private = { netmask = 26 connect_to_public_natgw = false } public = { netmask = 26 nat_gateway_configuration = "none" } } } module "vpc_from_cidr" { count = 1 source = "aws-ia/vpc/aws" version = "~> 4.4.2" name = local.vpc_name az_count = var.az_count cidr_block = local.vpc_cidr subnets = local.subnets } locals { vpc_id = module.vpc_from_cidr[0].vpc_attributes.id private_subnet_attributes = module.vpc_from_cidr[0].private_subnet_attributes_by_az public_subnet_attributes = module.vpc_from_cidr[0].public_subnet_attributes_by_az private_route_table_attributes = module.vpc_from_cidr[0].rt_attributes_by_type_by_az.private private_subnet_ids = [ for subnet_name, subnet_attributes in local.private_subnet_attributes : subnet_attributes.id if startswith(subnet_name, "private") ] } locals { # Construct a map of private/public subnets, private route tables & availability zones. # More: https://github.com/chime/terraform-aws-alternat/blob/91abdd8aedb701659e47d11bed0c15d049bde38d/variables.tf#L194-L202 # This relies on `data "aws_availability_zones"` to ensure Terraform knows # the size of the list during plan phase, otherwise it would have failed. # See: https://github.com/chime/terraform-aws-alternat/issues/104 vpc_config_for_alternat = [ for az in local.azs : { az = az public_subnet_id = local.public_subnet_attributes[az].id private_subnet_ids = [ for k, v in local.private_subnet_attributes : v.id if v.availability_zone == az ] route_table_ids = [ for k, v in local.private_route_table_attributes : v.id if startswith(k, "private/${az}") ] } ] } module "alternat_instances" { count = 1 source = "chime/alternat/aws" version = "0.6.0" nat_instance_type = "t4g.medium" lambda_package_type = "Zip" vpc_id = local.vpc_id vpc_az_maps = local.vpc_config_for_alternat } ```

If you'll find it useful then I can create a new example in the examples folder. ;)

[bwhaley]

Sure, an example showing how to use Alternat with aws-ia/vpc/aws would be excellent! Let's reorganize the examples/ directory to put the current example in a sub dir - let's call it "basic" - and another new dir for your example. Also need to update the path in the integration test.

[bwhaley]

I'm going to close this issue, but feel free to open a PR with the example if you want. Thanks for sharing your solution here.