search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
7.96k stars 967 forks source link

Dependency optimization broken since version 0.55.4 #3441

Open mickeder opened 4 days ago

mickeder commented 4 days ago

Describe the bug

Recently I've noticed that newer versions of Terragrunt run very slow when there are many recursive dependencies. This was not the case in the past, and I pinpointed the version where it changed to 0.55.4 - the introduction of Reference to dependency inputs.

According to the documentation about dependency block, there is a way to fetch only the outputs of the lowest level dependencies by using remote_state block. This way the outputs are not fetched from any higher level dependencies.

Unfortunately, this behavior was broken with the introduction of referencing dependency inputs in version 0.55.4. Currently all the outputs are fetched recursively for all dependency levels, which takes a long time when the dependency graph is complex but in reality only the lowest level dependency output is required for the root module.

Steps To Reproduce

Given below minimum configuration structure, I was able to reproduce the improper behavior.

# .
# ├── terragrunt.hcl
# ├── module-a
# │  ├── main.tf
# │  └── terragrunt.hcl
# ├── module-b
# │  ├── main.tf
# │  └── terragrunt.hcl
# └── module-c
#     ├── main.tf
#     └── terragrunt.hcl

# terragrunt.hcl
remote_state {
  backend = "local"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    path = "terraform.tfstate"
  }
}

# module-a/main.tf
output "test_output" {
  value = "hello from module-b"
}

# module-a/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}

# module-b/main.tf
variable "test_var" {
  type = string
}

output "test_output" {
  value = "hello from module-b"
}

# module-b/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}

dependency "module_a" {
  config_path = "../module-a"
}

inputs = {
  test_var = dependency.module_a.outputs.test_output
}

# module-c/main.tf
variable "test_var" {
  type = string
}

output "result" {
  value = var.test_var
}

# module-c/terragrunt.hcl
terraform {
  source = ".//"
}

include "root" {
  path = find_in_parent_folders()
}

dependency "module_b" {
  config_path = "../module-b"
}

inputs = {
  test_var = dependency.module_b.outputs.test_output
}

After applying module-a and module-b, I ran terragrunt plan for module-c with version 0.55.3, which gave the following debug logs:

time=2024-09-26T10:34:51+02:00 level=debug msg=Terragrunt Version: 0.55.3
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform --version prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=terraform version: 1.9.5
time=2024-09-26T10:34:51+02:00 level=debug msg=Reading Terragrunt config file at /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/terragrunt.hcl
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-a] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Getting output of dependency /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl for config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/terragrunt.hcl
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=[Partial] Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Detected module /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform output -json prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Retrieved output from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-b"
  }
}
 prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-b] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-09-26T10:34:51+02:00 level=debug msg=Included config /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-09-26T10:34:51+02:00 level=info msg=Downloading Terraform configurations from file:///home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c into /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=Copying files from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c into /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=Setting working directory to /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
time=2024-09-26T10:34:51+02:00 level=debug msg=The file path /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Generated file /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf. prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform init -no-color prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
Initializing the backend...
Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
time=2024-09-26T10:34:51+02:00 level=debug msg=Copying lock file from /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c/.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/.terraform.lock.hcl to /home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 
time=2024-09-26T10:34:51+02:00 level=debug msg=Running command: terraform plan -no-color prefix=[/home/michal/repos/bitbucket.org/trivecrd/devops/terragrunt/test/module-c] 

Changes to Outputs:
  + result = "hello from module-b"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Then I ran the same command with the newest Terragrunt 0.67.13, and the debug logs clearly show that outputs of module-a were fetched as well, even though they are not required by module-c.

10:33:42.580 DEBUG  Terragrunt Version: 0.67.13
10:33:42.580 INFO   Terragrunt Cache server is listening on 127.0.0.1:36205
10:33:42.580 INFO   Start Terragrunt Cache server
10:33:42.580 DEBUG  Provider cache dir "../../../../../../../.cache/terragrunt/providers"
10:33:42.581 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.581 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.581 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.581 DEBUG  Running command: terraform --version
10:33:42.609 DEBUG  terraform version: 1.9.5
10:33:42.609 DEBUG  Reading Terragrunt config file at ./terragrunt.hcl
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.609 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.609 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.610 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.610 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.611 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.611 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.613 DEBUG  [../module-a] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.613 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.614 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-b] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.614 DEBUG  [../module-b] Getting output of dependency ../module-a/terragrunt.hcl for config ../module-b/terragrunt.hcl
10:33:42.614 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.614 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] Did not find any locals block: skipping evaluation.
10:33:42.615 DEBUG  [../module-a] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.615 DEBUG  [../module-a] Detected module ../module-a/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory.
10:33:42.616 DEBUG  [../module-a] Running command: terraform output -json
10:33:42.645 DEBUG  [../module-a] Retrieved output from ../module-a/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-a"
  }
}
10:33:42.645 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.646 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
10:33:42.646 DEBUG  Getting output of dependency ../module-b/terragrunt.hcl for config ./terragrunt.hcl
10:33:42.646 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.646 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.647 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.647 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.648 DEBUG  [../module-b] Did not find any locals block: skipping evaluation.
10:33:42.648 DEBUG  [../module-b] [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.648 DEBUG  [../module-b] Detected module ../module-b/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory.
10:33:42.648 DEBUG  [../module-b] Running command: terraform output -json
10:33:42.678 DEBUG  [../module-b] Retrieved output from ../module-b/terragrunt.hcl as json: {
  "test_output": {
    "sensitive": false,
    "type": "string",
    "value": "hello from module-b"
  }
}
10:33:42.679 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.679 DEBUG  Did not find any locals block: skipping evaluation.
10:33:42.679 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
10:33:42.679 INFO   Downloading Terraform configurations from . into ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  Copying files from . into ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  Setting working directory to ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.680 DEBUG  The file path ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.
10:33:42.680 DEBUG  Generated file ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/backend.tf.
10:33:42.779 DEBUG  Discovered "registry.terraform.io" registry URLs: {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}
10:33:42.940 DEBUG  Discovered "registry.opentofu.org" registry URLs: {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}
10:33:42.941 INFO   Caching terraform providers for ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs
10:33:42.941 DEBUG  Running command: terraform init -no-color -no-color
10:33:43.016 STDOUT terraform: Initializing the backend...
10:33:43.016 STDOUT terraform: Initializing provider plugins...
10:33:43.016 STDOUT terraform: Terraform has been successfully initialized!
10:33:43.016 STDOUT terraform: You may now begin working with Terraform. Try running "terraform plan" to see
10:33:43.016 STDOUT terraform: any changes that are required for your infrastructure. All Terraform commands
10:33:43.017 STDOUT terraform: should now work.
10:33:43.017 STDOUT terraform: If you ever set or change modules or backend configuration for Terraform,
10:33:43.017 STDOUT terraform: rerun this command to reinitialize your working directory. If you forget, other
10:33:43.017 STDOUT terraform: commands will detect it and remind you to do so if necessary.
10:33:43.018 DEBUG  Running command: terraform init -no-color
10:33:43.061 STDOUT terraform: Initializing the backend...
10:33:43.061 STDOUT terraform: Initializing provider plugins...
10:33:43.061 STDOUT terraform: Terraform has been successfully initialized!
10:33:43.061 STDOUT terraform: You may now begin working with Terraform. Try running "terraform plan" to see
10:33:43.061 STDOUT terraform: any changes that are required for your infrastructure. All Terraform commands
10:33:43.061 STDOUT terraform: should now work.
10:33:43.062 STDOUT terraform: If you ever set or change modules or backend configuration for Terraform,
10:33:43.062 STDOUT terraform: rerun this command to reinitialize your working directory. If you forget, other
10:33:43.062 STDOUT terraform: commands will detect it and remind you to do so if necessary.
10:33:43.063 DEBUG  Copying lock file from ./.terragrunt-cache/gBB-mX3wXWUE19BSMdnf5F_Q6Oc/X6fQ3nxBTKZUZPrms2hMTKI5RMs/.terraform.lock.hcl to .
10:33:43.063 DEBUG  Running command: terraform plan -no-color
10:33:43.108 STDOUT terraform: Changes to Outputs:
10:33:43.108 STDOUT terraform:   + result = "hello from module-b"
10:33:43.108 STDOUT terraform: You can apply this plan to save these new output values to the Terraform
10:33:43.108 STDOUT terraform: state, without changing any real infrastructure.
10:33:43.108 STDOUT terraform: ─────────────────────────────────────────────────────────────────────────────
10:33:43.108 STDOUT terraform: Note: You didn't use the -out option to save this plan, so Terraform can't
10:33:43.108 STDOUT terraform: guarantee to take exactly these actions if you run "terraform apply" now.
10:33:43.109 INFO   Shutting down Terragrunt Cache server...
10:33:43.109 INFO   Terragrunt Cache server stopped

Expected behavior

Terragrunt should not fetch dependency outputs recursively, but just for the immediate dependency that is required. Basically the behavior from version 0.55.3 should be restored, but taking into consideration the newer feature of referencing dependency inputs as well.

Nice to haves

Versions

Additional context

We use Azure remote_state and dependency graphs as below are not uncommon. This causes the dependency fetching part take several minutes instead of several seconds.

dependency_graph

yhakbar commented 4 days ago

Hey @mickeder

Terragrunt does have to fetch dependencies recursively, as ancestor dependencies can influence outputs of immediate parents.

It should be caching the outputs of ancestor dependencies to minimize the performance penalty, however. Have you done a performance analysis to see what the impact has been in the latest version of Terragrunt?

You can take advantage of the OpenTelemetry integration to get more insight on that: https://terragrunt.gruntwork.io/docs/features/debugging/#opentelemetry-integration

mickeder commented 3 days ago

Hi @yhakbar, as I stated in my report, I tested it with the newest version of Terragrunt. It doesn't matter which version I use, after version 0.55.3 outputs of all ancestor dependencies are fetched, even though it was not the case before. The current behavior is also not reflected in the documentation that I mentioned (Can I speed up dependency fetching?), which states: If these conditions are met, terragrunt will only parse out the remote_state blocks and use that to pull down the state for the target module without parsing the dependency blocks, avoiding the recursive dependency retrieval.

In my opinion, the new feature implemented in version 0.55.4 giving the ability to reference inputs of dependencies broke the previous behavior and was not designed to change this in the first place.