search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
7.99k stars 970 forks source link

Improve caching for parsing configuration #2702

Open tomaciazek opened 1 year ago

tomaciazek commented 1 year ago

Describe the solution you'd like Implement config parsing cache also around ParseConfigString. Currently caching is implemented only for partial config parsing (PartialParseConfigString).

Describe alternatives you've considered Maybe it would be possible to reduce the number calls to parse config by other means. For instance, by disabling somehow the call of setIAMRole, which currently doubles the amount of times each file is re-parsed for files referenced by read_terragrunt_config in inlcuded files (through include block).

Additional context The setup in which this is most visible is when a file included using the include block reads another file using the read_terragrunt_config function. This results in re-parsing the file 14 times, even with --terragrunt-use-partial-parse-config-cache set. Commenting out the call to setIAMRole function in the config.go file reduces this amount to 8. I also noticed that removing expose = true cuts the number of parses by half too. Also, just having one or more references to the exposed variable increases the number of parses.

For larger projects, where also the dependency block is used, this quickly adds up to a long execution time. For instance, if we have modules A, B, C, and module A has dependency on both B and C, while module B also has dependency on C, the configuration file referenced by module C in setup as described above is read 23 times (instead of 14), provided that only module C includes that file.

tomaciazek-covantis commented 11 months ago

Any update on this one? This is starting to be a performance issue. With a deeper dependency graph (between root modules), and a couple of dozens of variables this results in tens of thousands of variable parsing executions (from what I can see in debug log). When running terragrunt <command> I need to wait for a long while before Terragrunt stops pre-processing and Terraform kicks in.

github-actions[bot] commented 2 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for raising this issue.