Dynamic variable names

[hendrikhalkow]

Hi there,

I would like to access variables by dynamically its name. For example, a variable called foo is accessed via var.foo, but I would like to access it via var["foo"]. This is useful for looping over variables: {for i in toset(["foo", "bar"]) : v => var[i]}.

Current Terraform Version

1.0.11

Use-cases

In a repo where we maintain the AWS organization with many AWS accounts, we want to enable users to maintain their account metadata, which is stored in a variable "metadata". However, we want to split it up into multiple auto.tfvars files so that we can assign different code owners to these files. So we create files $PROJECT_metadata.auto.tfvars that contain a variable called metadata. With this approach, the last one would win. When you name the variable $PROJECT_metadata instead, you also need to merge all $PROJECT_metadata variables into one metadata variable. Who would be code owner of that file?

Attempted Solutions

Our workaround is to loop over a list of YAML files in a sub-directory, which works, but it would be nicer to achieve that with HCL only.

Proposal

foo.auto.tfvars

foo = {
  "foo" = {}
}

bar.auto.tfvars

bar = {
  "bar" = {}
}

main.tf

variable "foo" {}

variable "bar" {}

output "foobar" {
  value = {for i in toset(["foo", "bar"]) : v => var[i]}
}

expected output:

foobar = {
  "foo" = {}
  "bar" = {}
}

actual output:

╷
│ Error: Reference to undeclared input variable
│ 
│   on  line 0:
│   (source code not available)
│ 
│ An input variable with the name "" has not been declared. This variable can be declared with a variable "" {} block.
╵
╷
│ Error: Invalid reference
│ 
│   on main.tf line 8, in output "foobar":
│    8:   value = {for i in toset(["foo", "bar"]) : v => var[i]}
│ 
│ A reference to a resource type must be followed by at least one attribute access, specifying the resource name.
╵
╷
│ Error: Invalid reference
│ 
│   on main.tf line 8, in output "foobar":
│    8:   value = {for i in toset(["foo", "bar"]) : v => var[i]}
│ 
│ The "var" object cannot be accessed directly. Instead, access one of its attributes.

References

https://github.com/hashicorp/terraform/blob/ab350289abb8fa4137e5a2c8fbcab32f887a310a/internal/addrs/parse_ref.go

[apparentlymart]

Hi @hendrikhalkow! Thanks for sharing this use-case.

Terraform relies on these static references in order to build a correct dependency graph between objects in the configuration, and so consequently var alone is not a symbol it its own right but rather just a prefix to help keep all of the input variables namespaced separately from other object types.

However, if you do wish to make dynamic decisions like this then you can use a local value to explain to Terraform that it ought to depend on both variables (to guarantee the correct execution order) but decide dynamically which of them to actually use:

locals {
  example = {
    foo = var.foo
    bar = var.bar
  }
}

Elsewhere in the module you can use local.example[expr] where "expr" is any expression producing a valid key from that mapping, to dynamically choose one of the values. The local value serves as an extra node in the dependency graph to allow depending on both of those variables (indirectly) at the same time, so that they will both be complete before choosing which one to select.

[nikolay]

@apparentlymart We keep hearing this over and over again - for dynamic provider configurations, etc. Why doesn't Terraform consider a pre-processing phase so that we don't have to use templates or Terragrunt? Is there one already for locals? I've been using your hack above, but it makes the code not DRY. All we need is what templating does, but done entirely in HCL, not using Jinja and other engines, which can produce a broken HCL and cannot be used with terraform-ls.

[apparentlymart]

I don't think I understand why building an appropriate data structure and then using that data structure would be considered a "hack"; this sort of approach is pretty typical in various programming languages where variable declarations are static rather than dynamic. (i.e. languages other than those where the evaluation context is totally dynamic, like Python or Lua.)

I also don't understand why this "makes the code not DRY". I don't see any repeated information here: the fact that there are variables called foo and bar is declared in one place, and the fact that there is a map containing both of those values is declared in one place. This local value declaration is new information that would not be inferable from anywhere else.

I don't really know how to continue this conversation because you seem to be expressing frustration about something broader than what this issue is about. If there is something specific you would like to be able to achieve in Terraform that you cannot, and you'd be willing to explain that use-case in enough detail that we could design solutions for it, I'd love to talk about it in a new issue. However, drastically redesigning the Terraform language to have a totally dynamic variable scope (like e.g. Python) or to interpolate values in the parsing phase (like e.g. Bash) is not on the table: those are fundamental design constraints such that changing them would amount to creating an entirely new language, and so if those capabilities are important to you then I would suggest using a product other than Terraform.

[nikolay]

@apparentlymart Because we already use it, and people always make the mistake of adding a new variable and forgetting to add it to the map in locals. Sometimes, they also make a typo (it happened last week!) in the key, which cannot be caught until you hit that case. Worst case scenario, you can build this hack in - always create a local called __variables__ and map all variables in it, and then when you use var[x], replace it local.__variables__[x].

[nikolay]

@apparentlymart Regarding the frustration: yes, it's been building slowly but steadily up with Terraform as the "C" letter in IaC is the most important one. I do understand the challenges you're dealing with (although I can't accept the "can't do" attitude) but Terraform needs to be more expressive and produce more reusable code. At my last job, for example, we had 1,500 Terraform plans! Anyway, all these things just add up and for people who use Terraform daily for at least two-thirds of their workday, the level of productivity loss and chasing hard-to-find bugs like those typos totals a substantial number of unproductive hours per month. In addition, we are a paid TFC client and as such, we can't even use Terragrunt, which exists for a number of good reasons - most TFC competitors support Terragrunt and you don't even support your own CDKTF, so, our hands are pretty tied up.