jq-style query to filter results

mjulianotq commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

The AWS CLI v2 has a --query parameter that takes a string for querying and filtering the JSON results of describe-* calls that is much more powerful than --filter. It would be nice to have similar functionality in the data sources in this Terraform provider. Unfortunately, what they have in the cli is not available in the SDK, but it should be fairly straightforward to do something similar if you use a jq parser like this one and construct queries from a query block.

New or Affected Resource(s)

All data sources that return more than one result.

Potential Terraform Configuration

data "aws_availability_zones" "azs" {
    state = "available"

    query {
        zone_name != "us-east-1a"
    }
}

This would run the equivalent of the following:

#!/bin/bash
$ aws ec2 describe-availability-zones --filter Name=state,Values=available | jq '.AvailabilityZones[] | select(.ZoneName!="us-east-1a")'

References

gdavison commented 3 years ago

Hi @mjulianotq, this is a great suggestion. I've done some research, and it looks like the for expression and for_each meta-argument introduced in Terraform 0.12 can do most if not all of this.

As with other "plural" data sources, aws_availability_zones returns only zone identifiers, and the "singular" data source aws_availability_zone needs to be called for details.

For example, you can do the following in us-west-2

provider "aws" {
  region = "us-west-2"
}

data "aws_availability_zones" "azs" {
  all_availability_zones = true
}

data "aws_availability_zone" "all" {
  for_each = toset(data.aws_availability_zones.azs.names)

  name = each.key
}

data "aws_availability_zone" "inline_filter" {
  # Exclude all "Wavelength" zones using the name
  for_each = toset([for s in data.aws_availability_zones.azs.names : s if length(regexall("-wlz-", s)) == 0])

  name = each.key
}

locals {
  # Exclude all "normal" AZs using the zone_type of the zone
  special_zones = [for az in data.aws_availability_zone.all : az if az.zone_type != "availability-zone"]
}

output "azs" {
  value = data.aws_availability_zones.azs.names
}

output "inline_filter" {
  value = data.aws_availability_zone.inline_filter
}

output "special_zones" {
  value = local.special_zones
}

The results are:

$ terraform plan

Changes to Outputs:
  + azs                   = [
      + "us-west-2-lax-1a",
      + "us-west-2-lax-1b",
      + "us-west-2-wl1-den-wlz-1",
      + "us-west-2-wl1-las-wlz-1",
      + "us-west-2-wl1-sea-wlz-1",
      + "us-west-2-wl1-sfo-wlz-1",
      + "us-west-2a",
      + "us-west-2b",
      + "us-west-2c",
      + "us-west-2d",
    ]
  + inline_filter         = {
      + us-west-2-lax-1a = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-lax-1"
          + id                     = "us-west-2-lax-1a"
          + name                   = "us-west-2-lax-1a"
          + name_suffix            = "lax-1a"
          + network_border_group   = "us-west-2-lax-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az2"
          + parent_zone_name       = "us-west-2a"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-lax1-az1"
          + zone_type              = "local-zone"
        }
      + us-west-2-lax-1b = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-lax-1"
          + id                     = "us-west-2-lax-1b"
          + name                   = "us-west-2-lax-1b"
          + name_suffix            = "lax-1b"
          + network_border_group   = "us-west-2-lax-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az4"
          + parent_zone_name       = "us-west-2d"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-lax1-az2"
          + zone_type              = "local-zone"
        }
      + us-west-2a       = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2"
          + id                     = "us-west-2a"
          + name                   = "us-west-2a"
          + name_suffix            = "a"
          + network_border_group   = "us-west-2"
          + opt_in_status          = "opt-in-not-required"
          + parent_zone_id         = ""
          + parent_zone_name       = ""
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-az2"
          + zone_type              = "availability-zone"
        }
      + us-west-2b       = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2"
          + id                     = "us-west-2b"
          + name                   = "us-west-2b"
          + name_suffix            = "b"
          + network_border_group   = "us-west-2"
          + opt_in_status          = "opt-in-not-required"
          + parent_zone_id         = ""
          + parent_zone_name       = ""
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-az1"
          + zone_type              = "availability-zone"
        }
      + us-west-2c       = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2"
          + id                     = "us-west-2c"
          + name                   = "us-west-2c"
          + name_suffix            = "c"
          + network_border_group   = "us-west-2"
          + opt_in_status          = "opt-in-not-required"
          + parent_zone_id         = ""
          + parent_zone_name       = ""
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-az3"
          + zone_type              = "availability-zone"
        }
      + us-west-2d       = {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2"
          + id                     = "us-west-2d"
          + name                   = "us-west-2d"
          + name_suffix            = "d"
          + network_border_group   = "us-west-2"
          + opt_in_status          = "opt-in-not-required"
          + parent_zone_id         = ""
          + parent_zone_name       = ""
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-az4"
          + zone_type              = "availability-zone"
        }
    }
  + special_zones         = [
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-lax-1"
          + id                     = "us-west-2-lax-1a"
          + name                   = "us-west-2-lax-1a"
          + name_suffix            = "lax-1a"
          + network_border_group   = "us-west-2-lax-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az2"
          + parent_zone_name       = "us-west-2a"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-lax1-az1"
          + zone_type              = "local-zone"
        },
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-lax-1"
          + id                     = "us-west-2-lax-1b"
          + name                   = "us-west-2-lax-1b"
          + name_suffix            = "lax-1b"
          + network_border_group   = "us-west-2-lax-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az4"
          + parent_zone_name       = "us-west-2d"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-lax1-az2"
          + zone_type              = "local-zone"
        },
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-wl1"
          + id                     = "us-west-2-wl1-den-wlz-1"
          + name                   = "us-west-2-wl1-den-wlz-1"
          + name_suffix            = "wl1-den-wlz-1"
          + network_border_group   = "us-west-2-wl1-den-wlz-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az3"
          + parent_zone_name       = "us-west-2c"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-wl1-den-wlz1"
          + zone_type              = "wavelength-zone"
        },
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-wl1"
          + id                     = "us-west-2-wl1-las-wlz-1"
          + name                   = "us-west-2-wl1-las-wlz-1"
          + name_suffix            = "wl1-las-wlz-1"
          + network_border_group   = "us-west-2-wl1-las-wlz-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az2"
          + parent_zone_name       = "us-west-2a"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-wl1-las-wlz1"
          + zone_type              = "wavelength-zone"
        },
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-wl1"
          + id                     = "us-west-2-wl1-sea-wlz-1"
          + name                   = "us-west-2-wl1-sea-wlz-1"
          + name_suffix            = "wl1-sea-wlz-1"
          + network_border_group   = "us-west-2-wl1-sea-wlz-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az1"
          + parent_zone_name       = "us-west-2b"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-wl1-sea-wlz1"
          + zone_type              = "wavelength-zone"
        },
      + {
          + all_availability_zones = null
          + filter                 = null
          + group_name             = "us-west-2-wl1"
          + id                     = "us-west-2-wl1-sfo-wlz-1"
          + name                   = "us-west-2-wl1-sfo-wlz-1"
          + name_suffix            = "wl1-sfo-wlz-1"
          + network_border_group   = "us-west-2-wl1-sfo-wlz-1"
          + opt_in_status          = "opted-in"
          + parent_zone_id         = "usw2-az2"
          + parent_zone_name       = "us-west-2a"
          + region                 = "us-west-2"
          + state                  = "available"
          + zone_id                = "usw2-wl1-sfo-wlz1"
          + zone_type              = "wavelength-zone"
        },
    ]

I'm going to close this issue. If these types of expressions aren't enough for your needs, it would be a better option to suggest the filtering improvements to Terraform as a whole, so that the entire practitioner community could benefit from them. To do that, please create an issue in the Terraform project.

mjulianotq commented 3 years ago

@gdavison Where does the singular/plural rule come from? There's a notable case where this provider invents a singular form of a lexically plural API call (that is to say, its name contains a plural, but it does not provide multiples of a singular resource as evidenced by the lack of a singular form in the API) that conceals the useful results of the API call and provides instead results for a nearly useless, unrelated use-case. The discussion of that case is obviously outside the scope of this ticket, but since you mentioned the singular/plural dichotomy, I'd like to ask if you can point me perhaps to a discussion where this design choice was made so I can better understand what benefit the team feels it provides users since my experience so far has suggested it was an arbitrary one that adds needless boilerplate to my config and encourages incorrect implementations of data sources which are only lexically plural.

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashicorp / terraform-provider-aws