search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
7.92k stars 966 forks source link

Terragrunt destroy of config deeply nested in config hierarchy checks dependencies of other modules many times #2981

Closed benwart-consensus closed 5 months ago

benwart-consensus commented 6 months ago

Describe the bug We have a deeply nested hierarchy of terragrunt config modules.

/.git
/terragrunt
/terragrunt/account
/terragrunt/account/env
/terragrunt/account/env/region
/terragrunt/account/env/region/namespace
/terragrunt/account/env/region/namespace/projects
/terragrunt/account/env/region/namespace/projects/stack
/terragrunt/account/env/region/namespace/pipelines
/terragrunt/account/env/region/namespace/projects/stack-iac
/terragrunt/account/env/region/namespace/projects/stack-build
...

When we try to destroy a stack the FindWhereWorkingDirIsIncluded() method is called which works its way from deepest to shallowest path searching recursively for all modules. This in turn checks the same modules multiple times. When there are lots of modules the repetitive loading alone (with no dependencies) takes almost an hour.

To Reproduce

  1. Create hierarchy of terraform configs (see folder structure above) with >3K stacks in the project/pipeline layer.
  2. Run terragrunt destroy from one of the project configs

Expected behavior When running versions older that 0.51.9 the job finishes in seconds rather than any newer build it takes almost an hour.

Versions

Underlying Problem After reviewing the code the issue is related to how the directories are scanned recursively repeatedly from lowest to highest checking for full dependencies of each module. This creates lots of redundancy, especially in our case, due to the fact that if there are no dependencies there is no storage of already checked modules to short circuit on.

Suggested Solution My opinion is the least intrusive option, but also likely a bit of a cheap hack, is to add a flag that allows skipping the check on destroy as a way of saying I'm good with the destroy so don't bother recursively checking for dependencies.

levkohimins commented 5 months ago

Resolved in v0.56.3 release.