Accept recursive calls to templatefile()

[Exchizz]

Terraform Version

Terraform v1.6.0-dev
on windows_amd64

Use Cases

We're going to use templatefile() for creating Azure Devops templates. For easier maintenance we want to be able to import templates from templates.

Attempted Solutions

main.tf

resource "local_file" "pipeline-templater" {
  content  = templatefile("pipeline.tftpl", { vars = local.variables })
  filename = "pipeline.yaml"
}

pipeline.tftpl

${ templatefile("template_header.tftpl",{vars = vars}) }
<pipeline goes here>

template_header.tftpl

pool: ${vars.agent_pool}

That gives the following error:

Error in function call; Call to function
│ "templatefile" failed: cannot recursively call templatefile from inside templatefile call..

Proposal

It is well documented behavior:

https://developer.hashicorp.com/terraform/language/functions/templatefile

The template may also use any other function available in the Terraform language, except that recursive calls to templatefile are not permitted. 

From the source code: https://github.com/hashicorp/terraform/blob/bc216caa65e0ca5e07dbfbbe5833790562091a5d/internal/lang/funcs/filesystem.go#LL69C1-L71C38

// As a special exception, a referenced template file may not recursively call
// the templatefile function, since that would risk the same file being
// included into itself indefinitely.

I don't think terraform should decided whether I can call templatefile() from a template or not. It works if I remove that check from the source code and compile terraform, so I suggest some mechanism should be implemented that informs the user, if the user has accidentally imported the same file info itself. Some recursion limit maybe ?

References

No response

[crw]

Thanks for this request!

[Exchizz]

@crw Any update on this ? Please let me know if I should create a PR :)

[crw]

@Exchizz, thanks for the great research in the bug description. Usually when there is a well-documented restriction in the Terraform language, it is due to considered trade-offs in the design. I will bring this up in triage but my guess is that a PR that changes this behavior would not be accepted given the intentional design decision to not allow recursion in templatefile.

[Exchizz]

@crw Hm, alright - thanks :)

What if it was an environment variable or an option to terraform that controls whether templatefile() can be called from the template ? Example:

TF_ALLOW_TEMPLATEFILE_FROM_TEMPLATE=true

Having this option to include files before running the templater would really improve the templating mechanism in terraform.

[birjj]

As someone also invested in this: would it be possible to clarify the reason for the design decision, if this is rejected?
From the comment in the source code it reads to me to as "there is a potential for edge cases entering infinite loops, therefore we disable the entire feature", which seems questionable. If that is the reason for the design choice, would a PR that implements recursion limits (or an explicit opt-in, as in @Exchizz's comment) be acceptable?

[Exchizz]

@crw Any update on this :) ?

[crw]

Yes, I brought this up in triage and the bottom-line feedback is that this issue would need more interest to be considered, particularly as it would mean revisiting a purposeful design decision. The code comment is a bit of a simplification, this was specifically engineered to be restricted to prevent more complicated design patterns ("programming with templates", as you might see with Jinja2 or Django). Even with the opt-in, the team would then need to support the more complicated logic of templates-calling-templates. Thus, needing to see more interest in such a design to justify the long-term support implications. We appreciate the feedback and of course will leave this request open to generate more feedback and use cases for this feature. Thanks!

[jasonwc]

I would love this feature for my use case, though I understand the reasoning to protect the current design.

In our case, we have a rather large startup script for one of our Google Cloud Compute instances:

# main.yml
resource "google_compute_instance" "some_compute_instance" {
  name                      = var.app_name
  # ...
  metadata = {
    startup-script               = templatefile("${path.module}/startup-script.tftpl", { var: var })
  }
}

It would be useful to extract pieces of the startup-script into their own templates for ease of maintainability. Then the startup-script would mainly compose those templates together.

# startup-script.tftpl
#! /bin/bash

# Setup authentication
${templatefile("authentication.tftpl", {var: var})}

# Add configuration
${templatefile("configuration.tftpl", {var: var})}

It's a simple use case and we can use other tooling or techniques to accomplish our goals, but I would still love to have a feature like this in Terraform.

[maycon]

Same situation here.

I'm using cloud-init to setup all my droplets without using any other tool.

In my case I have a boostrap.tftpl with the basic setup that must run if there's no specific setup for the given droplet.

  # droplets
  user_data = templatefile(
    fileexists("templates/${each.value.droplet_name}.tftpl")
    ? "templates/${each.value.droplet_name}.tftpl"
    : "templates/base/boostrap.tftpl", {
      username   = each.value.droplet_user
      keys_path  = "${var.config.keys_path}/${each.value.env_short_name}"
      name       = each.value.droplet_name
    }
  )

But I there is a specific template for the droplet (e.g. droplet-acme.tftpl) it must be the choose one, but not excluding the boostrap:

#cloud-config

${file("templates/base/boostrap.tftpl")}
${file("templates/base/swarm-worker-fw.tftpl")}

users:
  - name: spaces
    ssh_authorized_keys:
      - ${file("${keys_path}/id_rsa-labs-manager.pub")}

  - name: students
    ssh_authorized_keys:
      - ${file("${keys_path}/id_rsa-labs-worker.pub")}
...

I can try to use the old template_file data set to merge all files I need in a big template for each droplet, but for some reason it makes me feel uncomfortable.

Is there any workaround/best-practice to implement this kind of setup?

Cheers,

[mhmnemati]

@crw, Any update ?

[crw]

No new updates since my previous comment: https://github.com/hashicorp/terraform/issues/33272#issuecomment-1608051710d