Feature Request: Add function to evaluate string as HCL

[nicolas-lopez]

Use-cases and attempted solution

Hello I'm currently trying to standardize each of our IAM policies as a dedicated json file. And the main issue are terraform variable resolution inside those JSON using templatefile(). I feel it's a pain to pass a template map but I get the idea and the benefits from this method.

Therefor as I was just trying to pass all the variables of my project, I tried like dummy:

templatefile(myfile, var)

But I discovered there is actually no solution to access all variables at once (I though var would have been a map of all variables my bad). So finally had no other solution than using a map variable with all the variables used in each of my json policies (as templatefile accept map with more than required values this is a bit ugly but it works).

variable "templates" {
  type        = map
  default     = {
    foo: "bar"
  }
}

However I encountered a new problem, policies using resource attributes directly like ARNs. This is the point I choose to parse the policies dynamically in order to find all the terraform variables/resources in order to provide a map created on the fly with the required inputs for each policies.

locals {
  policies =
    [for file in fileset("policies", "*.json"):
      {policy = templatefile("policies/${file}", {for key in flatten(regexall("(\\$\\{.*?\\})", file("policies/${file}"))): trim(key, "}{$") => key if length(split(":", key)) == 1})}
    ]
}

In the code above the creation of all the variables on the fly is done by this part (more readable):

{for key in flatten(regexall("(\\$\\{.*?\\})", file("policies/${file}"))):
  trim(key, "}{$") => key if length(split(":", key)) == 1
}

This part give for example this output:

{
  "aws_iam_policy.mfa.name" = "${aws_iam_policy.mfa.name}"
  "var.account_id" = "${var.account_id}"
}

And this is where the main issue that led to this feature request. There is actually no way to evaluate a string to force interpolation and convert "${var.account_id}" to "00000000"!

Proposal

Add a function named eval(string) that takes a string and return an interpolated string.

Here is the doc for eval python built-in that does exactly what I propose and a bit more.

In my case I could update my code to:

locals {
  policies =
    [for file in fileset("policies", "*.json"):
      {policy = templatefile("policies/${file}", {for key in flatten(regexall("(\\$\\{.*?\\})", file("policies/${file}"))): trim(key, "}{$") => eval(key) if length(split(":", key)) == 1})}
    ]
}

And I would be able to load policies with any var dynamically on the fly and would be very happy =D

Thanks for your time!

[apparentlymart]

Hi @nicolas-lopez,

I've read the use-case you gave here a few times and I initially wasn't sure why you would not just pass in all of the named values that policies might use, and if a particular policy doesn't need a particular name then it'd just be ignored.

But on re-reading a few times, it seems more like what you are looking for is a way to evaluate a template using the module's own scope, not the local scope created by the attributes of the second argument to templatefile.

Unfortunately the same reason that motivates templatefile having its own separate variable scope also prevents the eval function you've proposed here: Terraform needs to analyze all of the references between resources and other objects before doing any other work, including expression evaluation, so that it can see what order to evaluate all of the objects.

templatefile achieves that because its second argument ends up containing all of the references for Terraform to analyze:

  example = templatefile("${path.module}/example.tmpl", {
    foo = var.foo # Static-analysis can see that this depends on var.foo
    bar = aws_instance.foo.id # Static-analysis can see that this depends on aws_instance.foo
  })

If the template file were allowed to contain a direct reference to var.foo or aws_instance.foo.id, without passing it explicitly in the second argument, Terraform would need to somehow analyze the function itself for dependencies, which is not something that is available in the current execution model: static analysis of dependencies treats the built-in functions as opaque.

That same constraint would therefore apply to a hypothetical eval function: it would need to take a second argument with a static definition of what values are available in its scope:

  example = eval("$${foo} and $${bar}", {
    foo = var.foo # Static-analysis can see that this depends on var.foo
    bar = aws_instance.foo.id # Static-analysis can see that this depends on aws_instance.foo
  })

The above therefore doesn't meet your use-case, because you'd still need to predict ahead of time what all of the possible references are. This static analysis step is a fundamental part of the Terraform language and not something any individual function can avoid.

With that said, perhaps we can peel back one level here and focus on the underlying problem to be solved, rather than the technical solution of dynamic references. Is there a reason why you can't statically define all of the variables that will be available for use in the templates, by writing out an explicit object constructor as in my two examples above?

[nicolas-lopez]

Hello,

I don't think you understood the use case and it's my fault because it's not that clear.

I want to create a terraform that will work without adding new resources for the IAM policies. That means I have a folder full of policies.json that I scan in the terraform.tf and I create each IAM policies using a single resource block with a for_each.

Regarding the fact that I

need to predict ahead of time what all of the possible references are.

I already do =) and I perform it because I have to use templatefile as there is no option that can interpolate variables inside a file with all the terraform scope which is totaly fine.

So what I do is I open each file scan vars inside it and then end up with a map like this

example = {
  "var.foo": "${var.foo}",
  "aws_instance.bar": "${aws_instance.bar}"
}

Given an input file like that:

# this is a var ${var.foo}
# this is a resource attribute ${aws_instance.bar}
# this is a var from IAM $${aws:username}

Unfortunately if I do:

templatefile("myfile", {"var.foo": "${var.foo}",  "aws_instance.bar": "${aws_instance.bar}"}}

where the strings "${var.foo}" and "${aws_instance.bar}" are crafted on the fly meaning it's not interpolated by terraform i would like to be able to trigger the interpolation myself.

Therefore that would work:

templatefile("myfile", {"var.foo": eval("${var.foo}"),  "aws_instance.bar": eval("${aws_instance.bar}")}}

If this is not clear enought here is my tf file and folder architecture maybe you will understand better. Folders:

├── backend.tf
├── groups.tf
├── policies
│   ├── cicd
│   │   └── ssm.json
│   └── mfa.json
├── policies.tf
├── provider.tf
├── tags.tf
├── terragrunt.hcl
└── variables.tf

Here is the policies.tf that contain all I describe as my use case:

resource "aws_iam_policy" "mfa" {
  name        = "mfa"
  policy      = file("./policies/mfa.json")
  description = "MFA enforcement allowing user to self manage it's MFA devices (cf. https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html)."
}

data "aws_caller_identity" "current" {}

locals {
  template    = merge(var.templates, data.aws_caller_identity.current)
  groups      = [for i in fileset("policies", "**"): dirname(i) if dirname(i) != "."]
  groups_list = flatten(
    [for group in local.groups:
      [for file in fileset("policies/${group}", "*.json"):
        {
          name   = format("${group}-%s", trimsuffix(basename(file), ".json")),
          path   = "/${group}/"
          policy = templatefile("policies/${group}/${file}", local.template)
          #policy = templatefile("policies/${group}/${file}", {for key in flatten(regexall("(\\$\\{.*?\\})", file("policies/${group}/${file}"))): trim(key, "}{$") => key if length(split(":", key)) == 1})
        }
      ]
    ]
  )
  groups_policy = zipmap(range(length(local.groups_list)), local.groups_list)
}

resource "aws_iam_policy" "customs" {
  for_each = local.groups_policy
  name     = each.value.name
  path     = each.value.path
  policy   = each.value.policy
}

You can see that in locals I scan for each subfolder of the folder "policies" in order to find json policies and define them in terraform.

The ultimate goal is that anyone can add a new json policy in any sub-folder of "policies" even create sub-folders and that will work. If the json contains a new value not already described in the terraform that means the var should be added to variables.tf and that people shouldn't have to touch any other terraform code.

Note that I currently use one of the solution you provided with this variable in the variable.tf:

variable "templates" {
  type        = map
  description = "Variables for IAM json policies interpolation templates (account_id, caller_arn, caller_user are predefined)"
  default     = {
  }
}

Thanks for the time you spend on this feature request let me know if you understand better my point of view and why I think eval() would be an awesome new feature for terraform.

If you want a demo of my problematic I will be please to share a few minutes in a call to make a demo.

[nicolas-lopez]

Hi let's up this topic again =) Bring us that eval() function !

[bergbrains]

While were at it, it would be nice to be able to submit a map or object as a value map. Also, a flag to allow unused values would be nice.

[derentis]

I would also like to see an eval function like @nicolas-lopez has suggested.

I've been using a replace function with regex so convert anything contained within {{}} to drop the moustache and instead be wrapped in ${} in a similar fashion as described here e.g. {{local.api_connection_key}} gets replaced with ${local.api_connection_key}. The string replacement is working as expected, but I've discovered that terraform isn't evaluating the resulting interpolation.

If someone has a workaround they could suggest in the absence of the proposed eval function being developed, I would be incredibly grateful.

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[apparentlymart]

Hi again! I apparently missed the responses here earlier, and I'm sorry about that.

Revisiting this issue again much later, I'm still a little unsure as to what the underlying goal was, but I can at least show a way to make the earlier example work: you need to give the templatefile function a data structure that matches the meaning of the references in the template, which means using nested objects instead of one object whose keys have dots:

templatefile("${path.module}/example.tmpl", {
  var = {
    foo = var.foo
  }
  aws_instance = {
    bar = aws_instance.bar
  }
})

A template file like the one shown would then evaluate as expected, because Terraform will interpret var.foo as a reference to the foo attribute of the object in variable var.

---

Arbitrary dynamic expression evaluation is a pretty marginal feature and so I don't think it's likely to be offered as a builtin, but in the meantime there are now some other options:

• The forthcoming Terraform v1.9 release will include a templatestring function that can render templates provided as strings. A template consistent entirely of a single interpolation sequence is effectively the same as evaluating the expression inside that interpolation sequence, so that function is effectively an eval function.

• Using the new possibility for provider-contributed functions I implemented my own provider hashicorp/hcl as a side-project (not an official HashiCorp project) and that exposes HCL expression evaluation as a Terraform function:

  terraform {
    required_providers {
      hcl = {
        source = "apparentlymart/hcl"
      }
    }
  }
  
  output "example" {
    value = provider::hcl::evalexpr("a + b", {
      a = 1
      b = 2
    })
  }

  This is HCL evaluation rather than Terraform evaluation, but since the Terraform expression language is an extension of HCL's it's sufficient for simple cases like these ones.

  Note that it has a design similar to templatefile where you need to provide an object representing the local scope to evaluate the expression in, for the same reasons I was describing above for Terraform's static analysis. There is no way to avoid that because Terraform's execution model relies on the ability to detect dependencies before evaluating any expressions.

Between those two possibilities I think we've got as close as we can reasonably get to dynamic expression evaluation, and so I'm going to close this issue. Thanks!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.