plan/apply of a module fetches state files for indirect dependencies (json rendering enabled)

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.

https://terragrunt.gruntwork.io/

MIT License

8.1k stars 984 forks source link

plan/apply of a module fetches state files for indirect dependencies (json rendering enabled) #3538

Open joaocc opened 2 weeks ago

joaocc commented 2 weeks ago

Describe the bug

When executing run-all plan (or apply) even with --terragrunt-ignore-external-dependencies, both package and state file of indirect dependencies is also being fetched. This causes execution time to increase vastly.

Consider a module my-app/terragrunt.hcl has dependencies for my-cluster/ and my-defs/, and my-cluster/terragrunt.hcl has dependency for my-vpc/. Executing a run-all plan on my-app/ should not need to install modules nor fetch state files of my-vpc/.

Expected behavior

Only fetch state of the direct dependencies.

Versions

Terragrunt version: v0.68.4
OpenTofu/Terraform version: v1.9.8
Environment details (Ubuntu 20.04, Windows 10, etc.): Debian 11

denis256 commented 2 weeks ago

Hello, try setting env variable TERRAGRUNT_STRICT_CONTROL="skip-dependencies-inputs", it will improve performance at cost of skipping dependency inputs

joaocc commented 1 week ago

Thanks. I was reading the docs and I am not sure if I am able to interpret correctly what this does.

Option 1: state files of direct dependency statements are not fetched (relying on local cache)
Option 2: state files of indirect dependency statements are not fetched
Other Thanks

yhakbar commented 8 hours ago

Hey @joaocc ,

We had a chat about this, and while the strict control that @denis256 mentioned does provide some optimizations for recursive fetching of dependencies it doesn't fully resolve the issue you're mentioning.

Terragrunt still inits in an ancestor dependency when it doesn't have to because of how it parses Terragrunt configurations (the relevant relatively inefficient code block is probably this one). While parsing the terragrunt.hcl file for a direct dependency to determine how to fetch outputs, Terragrunt ends up recursively traversing through ancestors to parse their configurations.

Preventing this behavior is going to take some adjustment in how we parse HCL configurations to do so safely, so please have patience as we look to resolve this issue. Of course, if you have your own thoughts on how to resolve this, you're welcome to submit a pull request!