Allow dynamically-recursive child module references with count and for_each

[willscripted]

Hi there tf team,

tl;dr, recursive modules generate an error on terraform init. ~Not sure if bug or feature request.~ Feature request.

module "xyz" {
  source = "./"
}

Terraform Version

Terraform v0.14.2

Terraform Configuration Files

Please see demo. Since it is an issue with modules, there are multiple files to show. The module definition at dir/main.tf#L4-10 is the section generating the error.

module "dir" {
  source = "./"
  for_each = { for file in local.root_filesystem : file.name => file if file.type == "dir" }

  name = each.key
  children = each.value.children
}

Debug Output

Crash Output

No crash.log found.

Expected Behavior

Successful terraform init.

Actual Behavior

Init failed, appears to be when stack depth exceeds some thresholds. (Props to whoever added that failsafe; it's much easier to track down when the process terminates.)

Steps to Reproduce

git clone git@github.com:willscripted/terraform-recursion-error.git
cd terraform-recursion-error
terraform init

Additional Context

Nothing special. Out-of-the-box hashicorp/terraform container.

References

VladRassokhin touches on it right at the end of #15276. His #22648 issue is closely related to this. The rest of an is:issue recursion search does not look relevant.

[apparentlymart]

Hi @willscripted! Thanks for opening this issue.

Self-referential modules are intentionally not supported, so framing this as a bug would mean we'd likely focus on making it fail in a more helpful way with a specific error message, rather than merely hitting a recursion limit.

With that said, I expect your intent here was to request the ability to have arbitrarily recursive modules where an empty count or for_each represents the termination condition. That is not something we have intended to support so far because the module tree is always static today: all modules are always present, and the only thing that varies is how many module instances they have associated.

I'm going to relabel this as an enhancement to represent the request for a change in behavior, but I'll also warn that there are some significant technical barriers to implementing this, because it would require a big change in Terraform's architecture so that either terraform init is able to evaluate for_each expressions (it currently evaluates no expressions at all) or Terraform uses a very different model for module installation. That means that there probably won't be any movement on it in the near future, because our focus is currently elsewhere, and that the next step for it would be to do some research and produce a technical proposal for how Terraform's architecture would change to accommodate this new requirement.

In the meantime though, I can see that there's a use-case implied by your report here: describing a filesystem tree of arbitrary depth, presumably with the goal of creating such a directory structure in a remote system. It'd help if you could add a follow-up comment with some more details on what you are doing there, both because that helps inform the design process and because we might be able to offer some other approaches you could use with Terraform today.

Likewise, if anyone else finds this enhancement request and has another use-case to share, please add a comment describing it and showing what you've tried so far and what didn't work. For a design change of this size we typically prefer to have several different use-cases to consider so we can consider multiple possible designs and see how well each one fits the various use-cases.

Thanks again!

[willscripted]

Really appreciate the quick (and thorough!) response, @apparentlymart

Self-referential modules are intentionally not supported, so framing this as a bug would mean we'd likely focus on making it fail in a more helpful way with a specific error message, rather than merely hitting a recursion limit.

For what it's worth, I think it's failing well. I'm just happy that it terminates at all -- and with an error to boot.

With that said, I expect your intent here was to request the ability to have arbitrarily recursive modules where an empty count or for_each represents the termination condition.

Yes, this exactly.

I was hoping for a magic flag or missing base condition; no such luck. I do hope it's possible someday! Any solution in terraform-proper feels cleaner than layering some tech over top.

A file system is the first analog I could think of that wouldn't use terraform resources. My actual use-case is a tree of aws_api_gateway_resource resources. If I want to route a GET request with path /x/y/z, I need to create an aws_api_gateway_resource for each of x, y, and z -- with each element as the child of the former. Then in that final z leaf, connect an aws_api_gateway_method resource of type GET.

I think my options now are a) write out terraform configuration to a fixed depth, or b) use a script to generate terraform configuration files out to arbitrary depth.

Anyways, thanks for the response and consideration!

[apparentlymart]

Thanks for the extra use-case detail!

For API gateway in particular, I've seen folks have success using Terraform to generate an OpenAPI schema to pass all together into the REST API resource, and thus avoid dealing with all of the other resource types. I'm not sure if that will help in your case, but mention it only in case it helps you get something going with today's Terraform.

[willscripted]

Oh, definitely. Thanks, i'll check it out!

[gtmtech]

@apparentlymart Just ran into this issue today.

Recursion would make my code so much nicer. I have to map out organisational trees in google folder structure and amazon organizations organizational_unit / organizational_account structure for companies with different designs on how they do it.

I want to describe the org in a simple yaml file e.g.

---
root:
  development:
    dev:
      team1:
        project1_1: name1
      team2:
        project2_1: name2
#---

etc. It would be so helpful to be able to instantiate primary folder nodes with the subfolder structure as an argument and let it recurse, rather than to write a set of iteration functions for each level at the terraform root and then instantiate each one and having to interpolate parents and childs along the way. The code for this looks so ugly in comparison.

So yeah, cloud account/project/subscription governance with arbitrary layouts is my usecase 👍

[gtmtech]

@apparentlymart One workaround I tried (which unfortunately didnt work) was to try this:

modules/
    folder
    folder2@ --> folder (symlink)
    folder3@ --> folder (symlink)
    folder4@ --> folder (symlink)

Then in the folder module:

variable "level" {
  type = string
  default = 1
}

Then in folder/subfolder.tf

module "subfolders" {
    for_each = toset(keys(var.subfolder_structure))
    source = "../folder${var.level + 1}"
}

│ Error: Variables not allowed
│ 
│ On ../../modules/folder/subfolders.tf line 3: Variables may not be used
│ here.

Shame, as this would have worked around the issue. Trying to think if there any other tricks on how it could be done without lots of copy+pasting of code

[ganniterix]

Recursion would make my code so much nicer. I have to map out organisational trees in google folder structure and amazon organizations organizational_unit / organizational_account structure for companies with different designs on how they do it.

I want to describe the org in a simple yaml file e.g.

I am trying to create OU's as well using some form of recursion, basically any kind of tree structure would need some kind of recursion. This would be extremely helpful.

[hartzell]

I’d like to assemble a set of GitLab groups/subgroups using the GitLab provider’s gitlab_group resource.

Subgroups are linked to their parent via the parent’s id, parent_id.

It’s straightforward to bang these out with an explicit set of resources, simply referring to the parent group and grabbing its ID in the subgroup.

I’m using a module to create the groups with appropriate defaults, it’s a riff and extension of ideas from Lotte-Sara Laan’s Module Parameter Defaults with the Terraform Object Type post. I can just add a separate invocation of my module for each resource, but I end up needing to call terraform init every time I add a new one, which seems unintuitive.

I’d rather just define the set of groups in a map variable and invoke the module once using a for_each loop.

The module is invoked something like this:

module "groups" {
  source         = "./modules/defaulted_gitlab_group"
  for_each       = local.our_groups
  name           = each.value["name"]
  path           = each.value["path"]
  description    = each.value["description"]
  parent_id      = each.value["parent_id"]
  group_settings = each.value["group_settings"]
}

Ideally our_groups would be something like this:

locals {
  # various things elided...
  our_groups = {
    "foo" : {
      name           = "Foo"
      path           = "foo"
      description    = "Foo"
      group_settings = local.group_defaults
      members = {
        "george-hartzell" = "owner",
      }
    }
    "foo-bar" : {
      name           = "Foo Bar"
      path           = "bar"
      description    = "Foo Bar"
      group_settings = local.group_defaults
      parent_id      = module.groups["foo"].gitlab_group.id
    }
  }
}

BUT, I end up with an error message like this (edited from a real example to match the sanitized input above):

╷
│ Error: Unsupported attribute
│
│   on foo_groups.tf line 35, in locals:
│   35:       parent_id      = module.groups["foo"].gitlab_group.id
│     ├────────────────
│     │ module.groups["foo"] is a object, known only after apply
│
│ This object does not have an attribute named "gitlab_group".
╵

I’ve also tried a second approach, where the local map contains the name of the parent group and I look it up in the module call/definition:

module "groups" {
  source         = "./modules/defaulted_gitlab_group"
  for_each       = local.our_groups
  name           = each.value["name"]
  path           = each.value["path"]
  description    = each.value["description"]
  group_settings = each.value["group_settings"]
  parent_id      = module.groups[each.value["parent_group_name"]].group.id
}

Somewhat predictably, this leads to a cycle:

╷
│ Error: Cycle: module.groups (close), module.groups.var.parent_id (expand), module.groups.gitlab_group.group, module.groups.output.group (expand)
│

The only other resource I found that supported a hierarchy is a Google folder but I was unable to find any examples of people generating them from a map like I’m trying to do.

I’d love suggestions for getting out of the corner I’ve painted myself into.

[AJCandfield]

@hartzell Same thing here. I would like to be able to manage Gitlab resources entirely with Terraform/Terragrunt but it's proving to be fairly difficult if I want to keep it DRY.

[wschult23]

I've encountered a similar problem. If you have nested maps, than you can't lookup the child attributes if you don't know how deep nested the attribute is. E.g. if you have a aws_instance object, then you can do the following:

lookup(aws_instance.my_instance,var.key)

if key is a top level attribute like "id", then it will return a proper value. But if the key is something like "tags.name" or "root_block_device.0.volume_size", it won't work. My first intention was, to use the split function and redo the lookup for each resulting value. But due to the lack of recursion capabilities, this won't work.

[kyparisisg]

@willscripted you can try and bounce between two or more modules which can give you the ability to emulate recursion assuming you call the modules within another module.

Just a wild thought...

[KyleKotowick]

@kyparisisg I've tried alternating between two modules, it has the same issue though (errors out at a certain depth).

[apparentlymart]

So far it seems like there are three ways people are trying to handle this sort of recursion:

• Using module calls themselves recursively with the intent of using a zero-instances call as the termination condition, as originally presented on this issue.

  This doesn't work because terraform init needs to decide the module tree statically in order to go fetch all of the modules. Terraform modules unfortunately often rely on modifying their own source code at runtime, using mechanisms like the archive_file data source or provisioners, and so it would be a breaking change to suddenly switch to sharing the same directory with multiple calls to the same module directory.

  Having some way for a module instance to explicitly request a temporary directory to work in and some way to bridge existing modules to make use of that (so that we don't break the ecosystem) could in principle combine with giving Terraform the ability to install each unique module package only once and refer to it many times, which could therefore in principle allow terraform init to resolve a self-recursive module dependency tree by terminating after it's installed all of the distinct packages.

  I think the big design question here is how exactly to get there from here, without requiring modifications to any existing modules.

• Preprocessing a recursive data structure into a flat one and then using that flat structure with a single module call.

  This sort of recursive-to-flat transformation is, as far as I can think of, not possible to do within the Terraform language itself because it lacks recursion capabilities as we discussed above. However, it is possible to implement escape hatches to code written in other languages by writing a provider, such as my experiment apparentlymart/javascript which embeds a small JavaScript interpreter for the express purpose of doing general computation that would normally be out of scope in the main Terraform language.

  Unfortunately this approach fails in any situation where different nodes in the tree need to refer to one another in a dynamic way, such as if the declaration of a child needs to point back to an opaque ID (generated by the server) of the parent. That means that the dependency graph ends up needing to be dynamic, which Terraform's execution model does not support -- Terraform always builds the dependency graph first and then visits the nodes of the graph in an order based on that, which means it must be clear what depends on what else before evaluating any other expressions.

• Preprocessing a recursive data structure into static Terraform configuration before running Terraform at all.

  This is a variant of the previous example which deals with the transformation outside of Terraform and just passes Terraform a generated flat set of resource blocks with relationships between them. This solves the problem from the previous option because the dependency graph is static as far as Terraform is concerned.

  CDK for Terraform is one way to get at this option, since it's a system expressly designed for generating Terraform configuration ergonomically from general-purpose languages. It does add some extra stuff into the workflow though, and typically means writing the entire configuration in a general-purpose language rather than just the small part that needed to be generated.

  A long time ago at my previous employer we did this using some relatively-simple Python scripts that our automation ran just before running Terraform, generating .tf.json files into a directory that already had hand-written .tf files and thereby allowing a mix of hand-written and generated in the same configuration.

The last of these is the only one that actually works today, and is what I would suggest for anyone who currently has use-cases of this sort. However, I think all three are worth exploring as possible ways to achieve a "built-in" answer; all three have some non-trivial design questions about how they would fit in to Terraform as it currently exists, without changing the meaning of any Terraform modules that were already written.

[kyparisisg]

@KyleKotowick I've solved this problem using CDKTF, and another way would be an engine which based on a payload it generates your terraform HCL (call to your modules with depends on). From working on both of the solutions, I'd definitely recommend the CDK for Terraform approach.