aws-ia / terraform-aws-vpc

AWS VPC Module
https://registry.terraform.io/modules/aws-ia/vpc/aws/latest
Apache License 2.0
82 stars 89 forks source link

Add more detailed notice about subnet calculation order #150

Closed fe-ax closed 1 month ago

fe-ax commented 2 months ago

I needed to figure out why my Terraform run gave an IP exhaustion error.

I ran the following Terraform code:

module "vpc" {
  source  = "aws-ia/vpc/aws"
  version = ">= 4.2.0"

  name                    = "${var.account_name}-vpc"
  vpc_ipv4_ipam_pool_id   = data.aws_vpc_ipam_pool.region_eu_central_1.id
  vpc_ipv4_netmask_length = 16
  az_count                = 3

  transit_gateway_id = data.aws_ec2_transit_gateway.my_tgw.id
  transit_gateway_routes = {
    pod-network = "0.0.0.0/0"
  }

  subnets = {
    pod-network = {
      name_prefix = "eks"
      netmask     = 18
      tags = {
        subnet_type = "private"
      }
    },
    control-plane = {
      name_prefix = "eks"
      netmask     = 28
      tags = {
        subnet_type = "private"
      }
    },
    transit_gateway = {
      netmask = 28

      tags = {
        subnet_type = "transit_gateway"
      }
    },
  }
}

Error:

│ Error: Invalid function argument
│ 
│   on .terraform/modules/vpc.vpc.calculate_subnets.subnet_calculator/main.tf line 6, in locals:
│    6:   addrs_by_idx  = cidrsubnets(var.base_cidr_block, local.networks_netmask_to_bits[*].new_bits...)
│     ├────────────────
│     │ while calling cidrsubnets(prefix, newbits...)
│ 
│ Invalid value for "newbits" parameter: not enough remaining address space
│ for a subnet with a prefix of 28 bits after 10.3.192.0/18.

Which seems to be caused by the order of subnet calculation. When changing the subnet to a smaller size it became clear why:

  subnets = {
    pod-network = {
      name_prefix = "eks"
      netmask     = 18
      tags = {
        subnet_type = "private"
      }
    },
    control-plane = {
      name_prefix = "eks"
      netmask     = 28
      tags = {
        subnet_type = "private"
      }
    },
    transit_gateway = {
      netmask = 28

      tags = {
        subnet_type = "transit_gateway"
      }
    },

Output:

      + cidr_block                                     = "10.3.0.0/29"
      + cidr_block                                     = "10.3.0.8/29"
      + cidr_block                                     = "10.3.0.16/29"
      + cidr_block                                     = "10.3.32.0/19"
      + cidr_block                                     = "10.3.64.0/19"
      + cidr_block                                     = "10.3.96.0/19"
      + cidr_block                                     = "10.3.128.0/29"
      + cidr_block                                     = "10.3.128.8/29"
      + cidr_block                                     = "10.3.128.16/29"

This happens because maps in Terraform are put in lexicographic order. Changing the network names using prefixes like 0 1 2 or a b c fixes this.