Add `--terragrunt-queue-include-units-reading` flag to Terragrunt

[yhakbar]

Summary

Add a new flag that works similarly to the terragrunt-modules-that-include flag, but instead adds Units (terragrunt.hcl files) to the queue of units that will be added to the Directed Acyclic Graph (DAG), if they read a file using a Terragrunt function.

NOTE: This RFC will use the relatively new language of "Unit" to refer to the directory where a terragrunt.hcl file leaf node is. See Stacks for more info. This language will become standard eventually.

Motivation

Users have asked for similar functionality to that which is available with terragrunt-modules-that-include, but specifically to handle things like updated JSON files and updated SOPS encrypted files to determine which Units need to be added to the DAG.

Right now, the best solution that users have available to them is to manually stitch together the list of Units that read external files by manually parsing terragrunt.hcl files. This can be error prone, however, as users have to work around issues like relative paths, and interpolated HCL functions changing which file is read.

Proposal

Introduce a hook that is used in every HCL function Terragrunt supports which reads a file to update a central mapping of files being read to the Units that read them.

Also introduce the flag --terragrunt-queue-include-units-reading that can be used in run-all commands to include Units that match up with the central mapping derived from the hook in HCL functions reading files.

Technical Details

Functions that definitely need the hook:

• read_terragrunt_config
• sops_decrypt_file
• read_tfvars_file

Over time, more functions can support this behavior.

Whenever any of these HCL functions are called, they will update the shared mapping of files to Units. As part of the run-all command, all Units in a given working directory are read and parsed, those parses should result in the mapping being updated, which will change how the queue is updated accordingly.

Press Release

Terragrunt can now watch for changes in external files!

Using the new --terragrunt-queue-include-units-reading flag, users can now drive updates to any Unit that reads from a file using an HCL function. This allows for CI/CD workflows to intelligently handle updates to files in formats other than HCL, like YML or JSON, including encrypted files using SOPS.

Drawbacks

This might result in a performance penalty for functions that read files.

We have to be careful introducing functionality like this, as functions often be repeatedly called during parsing in a way that degrades performance. Smart memoization, etc can reduce the impact of this.

Alternatives

1. Manually search through terragrunt.hcl files for references to files that read files in HCL functions.
2. Always wrap file reads with includes, so that terragrunt-modules-that-include can be used instead.

Migration Strategy

No response

Unresolved Questions

No response

References

No response

Proof of Concept Pull Request

No response

Support Level

• [ ] I have Terragrunt Enterprise Support
• [ ] I am a paying Gruntwork customer

Customer Name

No response

[josh-padnick]

I love the functionality here, but can we use a more intuitive flag name than --terragrunt-queue-include-units-reading?

Here are some random ideas:

• --teragrunt-include-dependent-files
• --teragrunt-include-related-files

[yhakbar]

@josh-padnick ,

Take a look at the rules being proposed under Technical Details --> Document Design Philosophy --> Rules.

The name of this flag is trying to do the following:

1. Make it clear what system the flag is impacting (Where are things being included? The answer is the runner queue).
2. Make it clear what the parameter for the flag is targeting (What is being included? The answer is units-reading the file specified as the value of the parameter).

This might feel more intuitive in the context of the --help of run as described in Technical Details --> Update help --> terragrunt run --help:

• --queue-exclude-dir value Unix-style glob of directories to exclude from the queue of Units to run. [$TERRAGRUNT_EXCLUDE_DIR]
• --queue-excludes-file value Path to a file with a list of directories that need to be excluded when running *-all commands. (default: .terragrunt-excludes) [$TERRAGRUNT_EXCLUDES_FILE]
• --queue-ignore-dag-order Ignore DAG order for --all commands. [$TERRAGRUNT_IGNORE_DEPENDENCY_ORDER]
• --queue-ignore-errors Continue processing Units even if a dependency fails. [$TERRAGRUNT_IGNORE_DEPENDENCY_ERRORS]
• --queue-exclude-external Ignore external dependencies for --all commands. [$TERRAGRUNT_IGNORE_EXTERNAL_DEPENDENCIES]
• --queue-include-dir value Unix-style glob of directories to include from the queue of Units to run. [$TERRAGRUNT_INCLUDE_DIR]
• --queue-include-external Include external dependencies for --all commands without asking. [$TERRAGRUNT_INCLUDE_EXTERNAL_DEPENDENCIES]
• --queue-include-units-including value Include Units that include the specified Unit. [$TERRAGRUNT_QUEUE_INCLUDE_UNITS_THAT_INCLUDE]
• --queue-strict-include Only Units under the directories passed in with '--queue-include-dir' will be included. [$TERRAGRUNT_QUEUE_STRICT_INCLUDE]

This will make all the flags that can impact the queue of Units that are going to be run prefixed with the same queue prefix so that they are found together in alphabetically sorted lists, like --help and docs. Users can also look at the list of flags they are using with the same stem of queue to work out the reasoning about how the system is impacted.

[yhakbar]

I cut a PoC alpha release here.

It adds a hook to read_terragrunt_config to automatically mark files read by the function as being detectable by the --terragrunt-queue-include-units-reading flag for inclusion into the Terragrunt run queue (all the units that are going to be run by Terragrunt during a run-all).

In addition, during conversations with the team, we discovered that there's advantage to being able to explicitly mark files as being read in this fashion, as they may be read by something hard to detect (e.g. read within a bash script, within OpenTofu or in an HCL function not directly defined in the Terragrunt codebase). For that reason, an additional HCL function will be introduced, mark_as_read which will serve to provide that capability explicitly when it's not convenient to handle it with native HCL functions.