Configure CLI options (dir, var file(s), state file) per-project/workspace

[OJFord]

Current Terraform Version

0.12.8

Use-cases

In my personal use of terraform, as well as professionally (where I didn't start the project), the terraform configuration goes in a sub-directory rather than the git root, from where I'd naturally be and run terraform commands.

In my personal use I have local state and var files inside a different directory, also in the project root. (It's a symlink to dir outside of git.)

Attempted Solutions

Makefile

I have a Makefile with -var-file, -state and DIR repeated in the combination taken by each terraform command I use, and then run make apply instead of terraform apply <long args>

The problems with this:

• annoying to use make instead of terraform (and easy to forget when I come back to it)
• doesn't work with e.g. taint, because I can't pass in what to taint

Env vars

TF_CLI_ARGS and TF_DATA_DIR is of course another option.

The problems with this:

• discovery
• multiple terraform projects

Using something like direnv to load/unload an env file, this becomes a more reasonable option, but still not ideal:

• we need a specific TF_CLI_ARGS_cmd for every cmd, since they take different combinations of -var-file, -state, and DIR

Proposal

I'd like to be able to set -var-file, -state, and terraform dir in a .terraform/conf.yaml, for example:

# project/.terraform/conf.yaml

var-file:
  - relative/to/.terraform
  - /absolute

var:
  specified: inline

# or just: `state: used/for/both`
state:
  in: relative/to/.terraform
  out: /absolute

dir: relative/to/.terraform

Though perhaps for big workspace users, per-workspace config would make more sense.

Alternatively, if these were options to the top-level terraform command (or UX-equivalently subcommands ignored unknown ones) then TF_CLI_ARGS='-var-file=x -state=y z' and TF_DATA_DIR=z/.terraform would do the job.

References

• 15934 discusses 'remembering' the directory from terraform init

[teamterraform]

Thanks for sharing these use-cases, @OJFord!

We've been thinking about some similar ideas recently... they're still too early and vague to share any details about right now, but it's helpful to have these pain points documented as another input into that discussion.

The high-level of the idea is similar to what you've thought about here, of formally separating the "configuration directory" from the "working directory" via one or more configuration files, but we're also considering how this would interact with workspaces particularly in an environment where some of them are remote workspaces (e.g. in Terraform Cloud). This train of thought is likely to change many core assumptions in Terraform today, and so isn't something we'll be able to march right into in the near future, but we'll hopefully continue to iterate on this idea and see if we can find a viable path forward that smooths the workflow for more complicated situations like what you've described here without drastically affecting those with more straightforward needs.

[OJFord]

Another problem with the environment variable approach is that:

TF_CLI_ARGS_plan='-state=statedir confdir'

means terraform plan no longer takes any arguments, since they should have preceded confdir. So, for example, -target=foo can't be used (without temporarily editing the environment variable).

The best approach today I think would be a shell function to wrap terraform, offering something similar to the TF_CLI_ARGS_*, except also passing argv in the appropriate place.

[OJFord]

-chdir is I think a recently added global option that helps here.

Unfortunately it can't be used with TF_CLI_ARGS, since it must come before the subcommand, i.e. terraform -chdir=subdir init, not, as TF_CLI_ARGS='-chdir=subdir' would render, terraform init -chdir=subdir.

[vitamindit]

As I cannot post in another ticket on the topic of global TF_CLI_ARGS, I am trying to come up with a solution for the configuration of global options via environment variables. A use case is described in the linked but now closed improvement (-chdir ..): https://github.com/hashicorp/terraform/issues/35418#issuecomment-2207068348

A good solution would be to provide two global Terraform environment variables analogue to the two global MAVEN environment variables. There is MAVEN_OPTS to set the options for the Maven process and MAVEN_ARGS to set global arguments for the maven reactor build. In addition to TF_CLI_ARGS, I can imagine TF_CLI_OPTS that do not refer to the sub-command but to the terraform runtime.

In addition to -chdir, TF_CLI_OPTS could also record other global options that do not relate to the sub-command. And the general configuration scheme would also be convergent with other industry standards such as maven.