functions to aggregate/summarize CIDRs

[tomalok]

Current Terraform Version

Terraform v0.12.20

Use-cases

For my particular use case, we pull a list of CIDRs that need to be routed a number of ways. The route tables have a maximum number of rules--which we are exceeding, unless we aggregate the long list of smaller CIDRs into a shorter list of larger CIDRs, which is currently being done by hand or by using online web tools. Having a Terraform function to accomplish this aggregation would be of great benefit.

Any "gaps" between CIDRs need to be preserved for our use case. However I imagine an option and/or second function to indicate "gaps are okay" or "gaps up to N size (or B bits) are okay" may also be useful in other situations.

Attempted Solutions

As mentioned above, our current solution is to aggregate the CIDRs by hand before inputting them to Terraform.

Proposal

• cidraggregate(<cidr-list>) => aggregated list of CIDRs, no gaps
• cidraggregate(<cidr-list>, <bits>) => aggregated list of CIDRs, allowing / gaps between
• cidrsummarize(...) as an alternate and/or additional (if "no gap" vs. "gap ok" needs to be separate) function name

References

n/a

[ewbankkit]

@tomalok The https://github.com/EvilSuperstars/go-cidrman project I started a while back and kind of ran out of steam on was aimed at eventually being used within Terraform somehow, either incorporated into a stand-alone provider with data sources or incorporation into https://github.com/moofish32/terraform-provider-cidr.

[apparentlymart]

Hi @tomalok! Thanks for sharing these use-cases.

I'd like to make sure I understand what you're looking for with some specific examples:

Input Expression	Result
cidraggregate(["10.1.0.0/17", "10.1.128.0/17"])	["10.1.0.0/16"]
cidraggregate(["10.1.0.0/17", "10.1.128.0/18", "10.1.192.0/18"])	["10.1.0.0/16"]
cidraggregate(["10.1.0.0/17", "10.1.128.0/18"])	["10.1.0.0/17", "10.1.128.0/18"]
cidraggregate(["10.1.0.0/17", "10.1.192.0/18"])	["10.1.0.0/17", "10.1.192.0/18"]
cidraggregate(["10.1.0.0/24", "10.1.1.0/24"])	["10.1.0.0/24", "10.1.1.0/24"]

If I were trying to describe the above succinctly in documentation, I guess I would describe this as:

cidraggregate finds the smallest set of CIDR address ranges covering all of the same IP address space as the given CIDR address ranges, eliminating any duplicates and combining multiple longer prefixes into a single shorter prefix where possible.

Is that the sort of thing you had in mind, or have I misunderstood?

One thing I'm feeling quite uncertain about is the difference between cidraggregate and cidrsummarize in your writeup. Was your intent that cidraggregate would return an error if it's not possible to reduce all of the given ranges into a single CIDR prefix, while cidrsummarize would behave like my table above and return potentially multiple results?

[tomalok]

@apparentlymart - yes, your examples and interpretation of what I'm asking for with cidraggregate is correct.

I would consider cidrsummarize (or the optional bits argument on cidraggregate) to be a "bonus" feature if it's feasible to implement without too much trouble and if its purpose is well-defined.

Perhaps a better way to do this is have a cidrminsize function to ensure a list of CIDRs and/or IPs be at least /bits in size. So...

cidrminsize(23, ["10.0.1.0/25", "10.0.2.0/24"])
["10.0.0.0/23", "10.0.2.0/23"]

...which could then be passed to cidraggregate and result in ["10.0.0.0/22"]

[AJD-UK]

This function would be very useful for where AWS S3 & DynamoDB public prefixes are placed into NACLs which have low quotas. These prefixes are often discovered using a data source and inserted into a NACL using count/for_each. e.g:

data "aws_ip_ranges" "s3" {
  regions  = [var.region]
  services = ["s3"]
}

AWS does not summarise its IP ranges as well as it could, as shown in this S3 example from eu-central-1

> data.aws_ip_ranges.s3.cidr_blocks
[
  "16.12.24.0/21",
  "16.12.32.0/22",
  "3.5.134.0/23",
  "3.5.136.0/22",
  "3.65.246.0/28",
  "3.65.246.16/28",
  "52.219.140.0/24",
  "52.219.168.0/24",
  "52.219.169.0/24",
  "52.219.170.0/23",
  "52.219.208.0/23",
  "52.219.210.0/24",
  "52.219.211.0/24",
  "52.219.218.0/24",
  "52.219.44.0/22",
  "52.219.72.0/22",
]

$ cat ip.py
import netaddr

with open('prefix.txt') as f:
    sublist = f.readlines()
sublist = [x.strip() for x in sublist]

print('There are ' + str(len(sublist)) + ' prefixes')

cidr = netaddr.cidr_merge(sublist)
print('IPs can be summarised into ' + str(len(cidr)) + ' prefixes')
print(cidr)

$ python ip.py
There are 16 prefixes
IPs can be summarised into 11 prefixes
[IPNetwork('3.5.134.0/23'), IPNetwork('3.5.136.0/22'), IPNetwork('3.65.246.0/27'), IPNetwork('16.12.24.0/21'), IPNetwork('16.12.32.0/22'), IPNetwork('52.219.44.0/22'), IPNetwork('52.219.72.0/22'), IPNetwork('52.219.140.0/24'), IPNetwork('52.219.168.0/22'), IPNetwork('52.219.208.0/22'), IPNetwork('52.219.218.0/24')]

[KylePeterDavies]

Hi, I agree with @AJD-UK that this would be useful. A vendor we are using publishes its IP Ranges as a JSON File. However, the CIDRs aren't summarised. This would also help with reducing the amount of rules in a Security Group (SG) or Network Access Control List (NACL) which often have Service Limits where they can only be increased by so many entries.

[yermulnik]

JFYI: while this hasn't been implemented natively with Terraform yet, we've employed https://registry.terraform.io/providers/amilevskiy/cidrblock/latest/docs to summarize CIDRs.

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[mxk]

This may be one of the worst things I've ever done in Terraform, but I really wanted to avoid a third-party provider, so I implemented a reasonably efficient, provider-free module for IPv4 CIDR aggregation (IPv6 is left as an exercise for the reader):

https://github.com/mxk/terraform-null-cidrsum