search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io
Other
43.13k stars 9.58k forks source link

Function to iterate over nested structures #27126

Open dee-kryvenko opened 3 years ago

dee-kryvenko commented 3 years ago

Current Terraform Version

v0.14.0

Use-cases

Often my modules receive complex data structures as an input (such as maps or objects). Here is a basic sample:

  teams = {
    team1 = {
      description = "awesome team"
    }
    team2 = {
      description = "another awesome team"
      members = {
        "someone" = {
          role = "maintainer"
        }
        "someone-else" = {}
      }
    }
    team3 = {
      description = "an average team"
      members = {
        "someone-else" = {}
      }
    }
  }

Let's say on the structure like this I need to create resources in a loop, for example:

resource "github_team_membership" "this" {
  for_each = <what?>

  team_id  = github_team.this[each.value.team].id
  username = each.value.member
  role     =  each.value.role
}

There's no easy way to iterate over such a multi-level structure, as for_each is expecting either a flat map or a set.

Attempted Solutions

Implementation for that could be something like:

locals {
  teams_members = merge([
    for team, team_cfg in var.teams : {
      for member, member_cfg in team_cfg.members == null ? {} : team_cfg.members : "${team}-${member}" => merge({
        team   = team
        member = member
      }, member_cfg)
    }
  ]...)
}

Proposal

It would be nice to have a one-line function to iterate over such a map. Something like iterable(map, fields...).

Something like

iterable({
  team1 = {
    members = {
      "foo" = {}
      "bar" = {}
    }
  }
  team2 = {
    members = {
      "bar" = {}
    }
  }
}, "members")

Should produce a:

{
  "team1-foo" = {
    keys = ["team1", "foo"]
    value = {}
  }
  "team1-bar" = {
    keys = ["team1", "bar"]
    value = {}
  }
  "team2-bar" = {
    keys = ["team2", "bar"]
    value = {}
  }
}

I'm not sure on the name for such a function, I'm using iterable as a placeholder. It should support an arbitrary number of fields. Also there should be an analogue function that would expect the last field in the chain to be just a list, i.e.:

iterable({
  team1 = {
    members = [
      "foo",
      "bar",
    ]
  }
  team2 = {
    members = [
      "bar"
    ]
  }
}, "members")

Should produce a:

{
  "team1-foo" = {
    keys = ["team1", "foo"]
  }
  "team1-bar" = {
    keys = ["team1", "bar"]
  }
  "team2-bar" = {
    keys = ["team2", "bar"]
  }
}

References

dee-kryvenko commented 3 years ago

Also function signature might need to be a iterable(map, separator, fields...) to use some other separator instead of - for keys. The keys in the resulting map needs to be predictable as potentially they might needs to be accessed to use resources output somewhere else.

apparentlymart commented 3 years ago

Hi @dee-kryvenko! Thanks for this feature request.

We've currently been recommending using merge as you showed, or alternatively flatten and setproduct for similar situations, with the ultimate goal of projecting the input data structure so that there is one element per needed resource instance.

Given the requirements you stated, if I didn't already see your example I expect I would've written it this way:

locals {
  team_members = flatten([
    for team_name, team in var.teams : [
      for member_name, member in team.members : {
        team_name   = team_name
        member_name = member_name
        role_name   = member.role
      }
    ]
  ])
}

resource "github_team_membership" "this" {
  for_each = {
    for m in local.team_members : "${m.team_name}-${m.member_name}" => m
  }

  team_id  = github_team.this[each.value.team_name].id
  username = each.value.member_name
  role     = each.value.role_name
}

If we can find some common patterns that come up a lot and simplify them then I'd definitely be interested in that, but so far I'm not really sure how to understand what you've proposed here as a general case vs. something specific to your particular module. I see your specific example but I'm not sure how describe in a general way what is a "field" and what isn't for the purposes of explaining the behavior of this function.

I'd like to explore more but I think we'd need to work through a few more examples first to see what use of this function might look like for different shapes of data structure, to make sure we're designing something that could be easy to understand for a future reader who might not already be familiar with the function. Understanding the problem better will hopefully also help us narrow down potential names for it that are descriptive of whatever problem this is aiming to address.

I think the main thing I'm getting caught up on reviewing your initial examples is that you specified members as a "field" and that caused the function to treat it differently than team1 or foo but I'm not really sure I understand the root problem that calls for treating "members" as special compared to the other levels, what sorts of data structures this function would and would not apply to, and how it would behave for more complex examples of that type of data structure.

I'm sorry I'm not yet familiar enough with the situation you're describing to ask more concrete questions. One question we could start with is whether the example I shared above would've also met your needs or if there was an additional requirement I didn't notice which we could talk about some more. However, if you can say anything else about possible general forms of the situation you're describing that will hopefully help me to understand what set of requirements we'd be aiming to address by introducing a new feature here.

Thanks!

dee-kryvenko commented 3 years ago

Hey @apparentlymart thanks for chiming in. You got it all right. I have everything I need, with the examples both you and me shared - there is nothing breaking currently and nothing I can't currently do.

I opened this issue because I think I can see a very common pattern here which is worth to have a one-line abstraction for.

The pattern is basically a multi-level (nested) map-like structure, for each branch of which we will need to create an instance of some resource. The github_team_membership resource was just an example to practically demonstrate the ask. It can be any types of resources that might be having parent-child relationship or a catalog-like structure. As someone who is writing terraform code daily - this is quite common thing to do. Often - multiple times within one single module.

I'm not sure iterable solution I proposed is the best one out there and there is probably some use cases I have missed, but I though just throwing it out there on the table so maybe collectively we can came up with something.

Let me try to abstract out the pattern that I think I'm seeing here. I would love to have a one-liner to convert the following structures into something flat so that terraform resources can iterate over via for_each (gonna use yaml for data structure representation just for the sake of readability):

folder1:
  sub-folder1:
    sub-sub-folder1: {}
    sub-sub-folder2: {}
  sub-folder2:
    sub-sub-folder1: {}
    sub-sub-folder2: {}
folder2:
  sub-folder1:
    sub-sub-folder1: {}
    sub-sub-folder2: {}
  sub-folder2:
    sub-sub-folder1: {}
    sub-sub-folder2: {}
---
folders:
  folder1:
    folders:
      sub-folder1:
        folders:
          sub-sub-folder1: {}
          sub-sub-folder2: {}
      sub-folder2:
        folders:
          sub-sub-folder1: {}
          sub-sub-folder2: {}
  folder2:
    folders:
      sub-folder1:
        folders:
          sub-sub-folder1: {}
          sub-sub-folder2: {}
      sub-folder2:
        folders:
          sub-sub-folder1: {}
          sub-sub-folder2: {}
---
folder1:
  sub-folder1:
    - sub-sub-folder1
    - sub-sub-folder2
  sub-folder2:
    - sub-sub-folder1
    - sub-sub-folder2
folder2:
  sub-folder1:
    - sub-sub-folder1
    - sub-sub-folder2
  sub-folder2:
    - sub-sub-folder1
    - sub-sub-folder2
---
folders:
  folder1:
    folders:
      sub-folder1:
        folders:
          - sub-sub-folder1
          - sub-sub-folder2
      sub-folder2:
        folders:
          - sub-sub-folder1
          - sub-sub-folder2
  folder2:
    folders:
      sub-folder1:
        folders:
          - sub-sub-folder1
          - sub-sub-folder2
      sub-folder2:
        folders:
          - sub-sub-folder1
          - sub-sub-folder2

Such structures can be of an arbitrary number of nesting levels - I did just 3 for demonstration purposes.

I was thinking about some examples in other languages but I couldn't come up with any. The closest probably is map functions from Ruby and Python. Which gives me another idea - probably the reason for this ask is because I can't implement DRY by myself as there is no easy way to plug-in set of custom functions from my private library of functions. I have to carry this sort of multi-line data transformation code across many many modules. Another thing is as I mentioned - I do this quite often, and because there's not an easy way to unit-test this data transformation (unless I decouple it to a separate module or something) or debug it - it sometimes getting very painful to transform something a little more complicated than these examples above.