produce map/object from nested for loop in terraform >0.12

[sleterrier]

Current Terraform Version

Terraform v0.12.6dev

Use-cases

While I found some examples on how to produce a list of maps, I am currently failing at producing a map of maps with a nested for loop.

How would you go about producing:

Outputs:

association-map = {
  "policy1" = "user1"
  "policy2" = "user1"
  "policy2" = "user2"
}

From:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

Attempted Solutions

I have tried many variations of:

locals {
  association-map = merge({
    for policy, users in var.iam-policy-users-map : {
      for user in users : {
        policy => user
      }
    }
  })
}

with zero success so far. I have only managed to get the following errors depending on the variation:

Error: Invalid 'for' expression. Extra characters after the end of the 'for' expression. Error: Missing attribute value. Expected an attribute value, introduced by an equals sign ("="). Error: Invalid 'for' expression. Key expression is required when building an object. Error: Missing key/value separator. Expected an equals sign ("=") to mark the beginning of the attribute value.

References

• #20288

Proposal

Assuming this is doable and I am just too dumb to figure it out (very likely), some documentation would be really helpful.

[teamterraform]

Hi @sleterrier , I refuse to believe that you are dumb - you're using terraform, after all! 😁You're also not mistaken.

Terraform 0.12 does not currently support nested for loops. You might find someone who can come up with a clever way to generate the structure you need in the community forum where there are more people ready to help, but I don't believe there's a direct function that will help here. Most of the functions work on maps, which can't have duplicate keys. It might also help to know more details of your use case, so we can see if there's another way to accomplish your goal.

I think there are two feature requests implicit in this issue: nested for expressions, and a function or functions that support creating objects (as opposed to maps, which as already said cannot have duplicate keys) from a for expression.

[sleterrier]

@teamterraform : Thank you for your answer! After sleeping on it and reading your comment, I realized I was stuck in a rabbit hole with no way out. We did prove I am dumb after all :)

Use-case

The use case - which should definitely have been described in the community forum instead of as an issue here - was to allow for local.association-map to be looped over in a _foreach resource. Even if an object with duplicate keys could be produced, it would therefore not have helped me.

Attempted Solutions

I wanted my team to be able to define GCP roles and members in two distinct lists to begin with:

variable admin_roles = {
  default = [
    "roles/resourcemanager.folderAdmin",
    "roles/resourcemanager.folderIamAdmin",
  ]
}

variable admin_members = {
  default = [
    "user:user1@example.com",
    "group:group1@example.com",
  ]
}

I then create a bindings map out of the two lists and feed it to a _google_folder_iambinding resource with _foreach:

locals {
  admin_bindings = {
    for role in var.admin_roles:
      role => var.admin_members
  }
}

resource "google_folder_iam_binding" "binding" {
  for_each = local.admin_bindings

  folder = "folder_1234"
  members = each.value
  role = each.key
}

Which works as expected and allows me to remove/add members and roles anywhere in the starting lists without triggering a delete/create of the resulting resources. But I then realized I did not always want to be authoritative on the roles bindings, so I embarked on trying to produce a map I could feed to a _google_folder_iammember resource with _foreach:

# !! We established this is not possible !! Don't try this at home.
locals {
  local.admin_bindings_additive = merge({
    for role, members in local.admin_bindings : {
      for member in members : {
        role => member
      }
    }
  })
}

resource "google_folder_iam_member" "member" {
  for_each = local.admin_bindings_additive

  folder = "folder_1234"
  member = each.value
  role = each.key
}

But failed miserably, for obvious reasons now. I have to go back to the drawing board and re-think the whole approach...

Thanks again, and please let me know if you'd rather have me move this thread to the forum so we do not pollute the issues section.

[teamterraform]

No worries! I think your specific case is worth posing to the community forum, but (against all odds) this is the first request for nested for expressions, so we should keep it open as an enhancement.

[sleterrier]

Reference

There is however one last thing I am not clear about, based on #20288. I thought nested for loops were already supported given that:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

locals {
  association-list = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        user   = user
        policy = policy
      }
    ]
  ])
}

output association-list {
  value = local.association-list
}

would actually produce a list of maps:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

association-list = [
  {
    "policy" = "policy1"
    "user" = "user1"
  },
  {
    "policy" = "policy2"
    "user" = "user1"
  },
  {
    "policy" = "policy2"
    "user" = "user2"
  },
]

[mildwonkey]

HI @sleterrier! I'm the one who make the erroneous statement about nested loops, sorry! Let me spend a little longer thinking about this - I'm still fairly positive that we can't do precisely what you are requesting at this time, but I might be able to come up with a workaround, or at least a better-thought-out feature request :) Sorry again, cheers and thanks for the update.

[mildwonkey]

Would a list work, in place of a map? You might have to tweak the format output, but I got this:

Outputs:

association-list = [
  "policy1 = user1",
  "policy2 = user1",
  "policy2 = user2",
]

config:

locals {
  association-list = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : [
        format("%s = %s", policy, user)
      ]
    ]
  ])
}

[sleterrier]

Hi @mildwonkey , thanks for taking a stab at this, much appreciated!

I am however in dire need of a map, so I can create multiple _google_folder_iammember resources by iterating over its keys with _foreach. The end goal is to avoid the resulting resource(s) to be tied to a list index, but created with a unique name instead. So any elements of var.iam-policy-users-map could be removed/added without triggering a delete/create of any other elements (hopefully this sentence makes sense).

I am still wondering how the association-list code you pasted above does not constitute a nested for loop though. And how come we could not use a similar logic to create a map of maps? (instead of a list of maps in my example or a list of strings in your example). But we already proved I am dumb, so that could just be that.

[mildwonkey]

It is a nested loop! The error was mine, I misstated. Nested loops do indeed work.

One issue with what you're trying to do is that this is an invalid map. A map, by definition, does not allow for duplicate keys, and you have "policy2" twice:

association-map = {
  "policy1" = "user1"
  "policy2" = "user1"
  "policy2" = "user2"
}

Map keys have to be unique.

[sleterrier]

Oh, got it now, the different usernames confused me :)

I did realize my fist attempt was foolish because of the duplicate keys it would have produced. Even if such an object could have been created, it would not have helped me iterate over it with _foreach. I am however still interested in seeing how one could produce a map (of, let's say maps) in nested for loops?

[sleterrier]

To clarify, here is the new question. Would there be a way to produce:

Outputs:

association-map = {
  "policy1_user1" = {
    "policy1" = "user1"
  }
  "policy2_user1" = {
    "policy2" = "user1"
  }
  "policy2_user2" = { 
    "policy2" = "user2" 
  }
}

From:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

If that's doable and unless I am missing something again, I believe that would allow for what I have in mind with for_each.

[mildwonkey]

It is possible to create a nested map of maps. The problem is that you need the map key at the top level.

This is (obviously) a useless structure, but I wanted to illustrate that it is possible:

locals {
  association-list = {
    for policy, users in var.iam-policy-users-map:
      policy => {      // can't have the nested for expression before the key!
        for u in users:
           policy => u...
      }
  }
}

Outputs:

association-list = {
  "policy1" = {
    "policy1" = [
      "user1",
    ]
  }
  "policy2" = {
    "policy2" = [
      "user1",
      "user2",
    ]
  }
}

Your example is not directly possible using for expressions because you don't have all the information you need to format the 'key' before stepping into the nested loop, ie $policy_$user

This brings us back around to my earlier response: there may very well be a workaround to get the result you are looking for, but nothing direct nor obvious. Still, a good question for the community forum! Now that we've clarified my earlier mistake (re nested loops), I am going to remove the enhancement label.

No one here is dumb, even me (and I was feeling pretty dumb earlier, which is why I switched to my personal account to own up to the mistake). Sometimes we might sound or feel dumb, but we aren't dumb. I promise :)

[sleterrier]

Thank you very much for your answers @mildwonkey , that was really helpful. I had made changes to the expected output on my previous comment in the meantime - a bad edit habit of mine - which I reverted to keep this thread intelligible. I will take it to the forums from here if needed.

I believe the original intent of my issue remains though: some documentation on nested loops would be really helpful.

[sleterrier]

@apparentlymart gave a great answer in the community forum in case of anyone else was interested. Thanks again for all your help, much appreciated!

[arcotek-ltd]

I asked a similar question is SO: https://stackoverflow.com/questions/58343258/iterate-over-nested-data-with-for-for-each-at-resource-level

Although the answer solved the problem at the time, I've now adjusted the data set meaning I need to traverse three tiers.

[markchalloner]

This can be reduced down to a single variable:

variable "instance_types" {
  type = list(string)
  default = ["c5-2xlarge", "r5-4xlarge"]
}

locals {
  availability_zones = ["a", "b", "c"]
  # Create a map of availability zones and instance_types maps:
  # { ..., "c5-2xlarge-"b"" = {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ...,
  #   ..., "r5-4xlarge-"b"" = {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... }
  instance_types_availability_zones = {
    for instance_type_availability_zone in
      # Flatten the list of lists of availability zones and instance_types maps into a single list:
      # [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ...,
      #   ..., {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... ]
      flatten(
        # Create a list of lists of availability zones and instance_types maps per instance type:
        # [
        #   [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ... ],
        #   [ ..., {"availability_zone" = "b", "instance_type" = "r5-4xlarge"}, ... ]
        # ]
        [
          for instance_type in var.instance_types: [
            # Create a list of maps of the availability zone and instance_type:
            # [ ..., {"availability_zone" = "b", "instance_type" = "c5-2xlarge"}, ... ]
            for availability_zone in local.availability_zones: {
              availability_zone = availability_zone
              instance_type = instance_type,
            }
          ]
        ]
    ):
      "${instance_type_availability_zone.instance_type}-${instance_type_availability_zone.availability_zone}"
        => instance_type_availability_zone
  }
}

output "instance_types_availability_zones" {
  value = local.instance_types_availability_zones
}

[abdrehma]

My crack at it:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

output "association-map" {
 value = flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        policy = user
      }
    ]
  ])
}

Outputs:

association-map = [
  {
    "policy" = "user1"
  },
  {
    "policy" = "user1"
  },
  {
    "policy" = "user2"
  },
]

[mhumeSF]

@abdrehma The output association-map produces a value not consumable by for_each.

It'd need to be something like transforming

default = {
  "policy1" = [ "user1" ]
  "policy2" = [ "user1", "user2" ]
}

to

{
  "policy1" = "user1",
  "policy2" = "user1",
  "policy2" = "user2",
}

But this won't work since map keys are unique?

[abdrehma]

@mhumeSF Not pretty, but here's a quick example of turning that list of tuples into a usable map:

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

locals{
 association-list =  flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        "${policy}-${user}" = {
          "user" = user
          "policy" = policy
          }
      }
    ]
 ])

 association-map = { for item in local.association-list: 
     keys(item)[0] => values(item)[0]
   }
}

output "association-map" {
   value = local.association-map
}

Output:

association-map = {
  "policy1-user1" = {
    "policy" = "policy1"
    "user" = "user1"
  }
  "policy2-user1" = {
    "policy" = "policy2"
    "user" = "user1"
  }
  "policy2-user2" = {
    "policy" = "policy2"
    "user" = "user2"
  }
}

Example for_each usage:

resource "null_resource" "echo" {
  for_each = local.association-map
  provisioner "local-exec" {
    command = "echo 'policy - ${each.value.policy}, user - ${each.value.user}'"
  }
}

[mhumeSF]

@abdrehma Clever creating the unique key

[skgbox]

As an workaround the following code works for me:

locals { routes_iteration = flatten([ for route_table_id in local.route_tables_all_ids_svc: [ [ for cidr in [local.vpc_cidr_dev, local.vpc_cidr_tst, local.vpc_cidr_prd]: "${route_table_id}#${cidr}" ] ] ]) }

resource "aws_route" "svc_to_apps" { provider = aws.svc for_each = toset(local.routes_iteration) route_table_id = split("#", each.value)[0] destination_cidr_block = split("#", each.value)[1] transit_gateway_id = aws_ec2_transit_gateway.ad-router.id }

[sandy724]

BTW there is a hackish solution for nested loop

locals {
  rules = {
    rule_list = [
      {
        from_port    = 2181
        to_port      = 2181
        protocol     = "http"
        cidr         = []
        source_SG_ID = ["abc", "def", "ghi"]
      },
      {
        from_port    = 2181
        to_port      = 2181
        protocol     = "http"
        cidr         = []
        source_SG_ID = ["def"]
      }
    ]
  }

  rule_list1 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[0]
    }
  ]
  rule_list2 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[1]
    } if length(ruleList.source_SG_ID) > 1
  ]
  rule_list3 = [
    for ruleList in local.rules.rule_list:
    {
        from_port = ruleList.from_port
        to_port = ruleList.to_port
        protocol = ruleList.protocol
        cidr = ruleList.cidr
        source_SG_ID = ruleList.source_SG_ID[2]
    } if length(ruleList.source_SG_ID) > 2
  ]
  newRules = {
    rule_list = concat(local.rule_list1,local.rule_list2,local.rule_list3)
  }
}

output "rules" {
   value = local.rules.rule_list[0].source_SG_ID
}

output "newRules" {
   value = local.newRules
}

Don't bash me for this hack, but it meets my requirement

[melck]

Any news on this feature? Do you any plan to add it ?

I'm really interested too. At the moment, I use hack way like others or i change my data structure when i can.

Overall, it is really difficult to read / understand unlike nested expressions. I run the risk of losing my colleagues and dissuading them from using Terraform on a complex project.

[avishnyakov]

@abdrehma, thanks a lot for sharing this handy approach 🚀

• https://github.com/hashicorp/terraform/issues/22263#issuecomment-581205359

Adopting slightly modified version originally posted by @abdrehma . By composing key/value, we can have better readability and cleaner syntax for final for-each.

• make "${policy}-${user}" as key property
• wrap the rest into value property, anything you want
• ^ which makes association-map cleaner with item.key => item.value syntax

variable iam-policy-users-map {
  default = {
    "policy1" = [ "user1" ]
    "policy2" = [ "user1", "user2" ]
  }
}

locals{
 association-list =  flatten([
    for policy, users in var.iam-policy-users-map : [
      for user in users : {
        key = "${policy}__${user}"
        value = {
          "user" = user
          "policy" = policy
          }
      }
    ]
 ])

 association-map = { for item in local.association-list: 
     # key/value refers to your structure in the nested for-each early
     # just cleaner and easier to understand for other people
     item.key => item.value
   }
}

output "association-map" {
   value = local.association-map
}

[LevonBecker]

@abdrehma The output association-map produces a value not consumable by for_each.

It'd need to be something like transforming

default = {
  "policy1" = [ "user1" ]
  "policy2" = [ "user1", "user2" ]
}

to

{
  "policy1" = "user1",
  "policy2" = "user1",
  "policy2" = "user2",
}

But this won't work since map keys are unique?

From the TF console this works. TF 1.0.8. Jetbrains IDE HCL plugin isn't happy with the syntax though. It's forcing it to read the variable and not assume a string literal.

locals {
  test_map = {
      policy1 = [ "user1" ]
      policy2 = [ "user1", "user2" ]
  }
}

flatten([for policy, users in local.test_map : [ for user in users : { "${policy}" = user } ] ])

[
  {
    "policy1" = "user1"
  },
  {
    "policy2" = "user1"
  },
  {
    "policy2" = "user2"
  },
]

[exabugs]

@abdrehma 'merge' is convinient in this case.

variable "iam-policy-users-map" {
  default = {
    "policy1" = ["user1"]
    "policy2" = ["user1", "user2"]
  }
}

locals {
  association-map = merge([
    for policy, users in var.iam-policy-users-map : {
      for user in users :
        "${policy}-${user}" => {
          "user"   = user
          "policy" = policy
        }
    }
  ]...)
}

output "association-map" {
  value = local.association-map
}

Output:

association-map = {
  "policy1-user1" = {
    "policy" = "policy1"
    "user" = "user1"
  }
  "policy2-user1" = {
    "policy" = "policy2"
    "user" = "user1"
  }
  "policy2-user2" = {
    "policy" = "policy2"
    "user" = "user2"
  }
}

Example for_each usage:

resource "null_resource" "echo" {
  for_each = local.association-map
  provisioner "local-exec" {
    command = "echo 'policy - ${each.value.policy}, user - ${each.value.user}'"
  }
}

[odg0318]

@exabugs Huge thanks. Your solution is the best here. I can't imagine to use ... and merge. Perfect!

[jefferson-monteiro-iupp]

I have a question similar to this one, but it's for creating folders on S3. I will have some nested folders with more than two levels like below:

Bucket-1

• provider-1
  • batch-process
    • people
    • cart
  • online-process
    • log
    • order ...

I managed to create a list with all the S3 directories I would like to have, but I know this is not the most efficient way.

I tried to follow some examples found here but they returned an error. What is the correct way to do this iteration without having to write the entire directory?

[crw]

Hi @jefferson-monteiro-iupp, would you please try asking your question in the community forum? This helps keep the issue conversation on the topic of the original issue. If you are facing a new or different issue, feel free to open a new report following the issue report template. Thanks!

[gentitope]

Someone help me to flatten this:

variable vpc_peer {
  default = {
    vpc_peer1 = {
      aws_account_id = "acct-id"
      region  = "region-1"
      vpc_id  = "vpc-id"
      route_table_cidr_block = "10.0.0.0/16"
      route_table_ids = [
         "rtb-1",
         "rtb-2"
      ]
    }
  }
}

Expectation:

With for_each function

 resource "aws_route" "my_route" {
      destination_cidr_block    = "172.0.0.0/24"
      id                        = (known after apply)
      instance_id               = (known after apply)
      instance_owner_id         = (known after apply)
      network_interface_id      = (known after apply)
      origin                    = (known after apply)
      route_table_id            =  ["rtb-1", "rtb-2"]
      state                     = (known after apply)
      vpc_peering_connection_id = (known after apply)
    }

Thanks

[gentitope]

i got it solved!

[angeladojchinovska-allocate]

I have a question similar to this one, but it's for creating folders on S3. I will have some nested folders with more than two levels like below:

Bucket-1

• provider-1

  • batch-process

  • people

  • cart

  • online-process

  • log

  • order ...

I managed to create a list with all the S3 directories I would like to have, but I know this is not the most efficient way.

I tried to follow some examples found here but they returned an error. What is the correct way to do this iteration without having to write the entire directory?

Any updates on this?