search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
7.91k stars 966 forks source link

Issues when Excluding Dependency Dirs #1605

Open robarthur opened 3 years ago

robarthur commented 3 years ago

Terraform version: Terraform v0.14.2 Terragrunt version: terragrunt version v0.28.7

I'm seeing unexpected behaviour when I try to use the following flags with either a run-all or a destroy-all:

--terragrunt-exclude-dir
--terragrunt-ignore-external-dependencies

Specifically I can't prevent dependencies that are outside of my current working directory from being included in *-all commands.

I have a setup that looks like this:

aws/customer-account/vpc/
aws/customer-account/transit-gateway-attachment/

aws/networking-account/vpc/
aws/networking-account/transit-gateway/

The customer account transit-gateway-attachment has dependencies on the networking account VPC and Transit Gateway. But when if I run a terragrunt run-all destroy from the aws/customer-account/ directory, I don't want to destroy any resources in the networking account.

What I tried (some info modified/redacted...):

ubuntu@machine:/aws/customer-account/$ terragrunt run-all destroy --terragrunt-ignore-external-dependencies
INFO[0000] Stack at  /aws/customer-account:
  => Module  /aws/networking-account/transit_gateway (excluded: false, dependencies: [ /aws/networking-account/])
  => Module  /aws/networking-account/ (excluded: false, dependencies: [])
  => Module  /aws/customer-account /transit_gateway_attachment (excluded: false, dependencies: [ /aws/networking-account/transit_gateway,  /aws/customer-account /vpc])
  => Module  /aws/customer-account /vpc (excluded: false, dependencies: []) 
WARNING: Are you sure you want to run `terragrunt destroy` in each folder of the stack described above? There is no undo! (y/n) n
ubuntu@machine:/aws/customer-account$ terragrunt run-all destroy --terragrunt-exclude-dir */networking-account/*
Module /aws/customer-account/transit_gateway_attachment depends on module /aws/networking-account/transit_gateway, which is an external dependency outside of the current working directory. Should Terragrunt run this external dependency? Warning, if you say 'yes', Terragrunt will make changes in /aws/networking-account/transit_gateway as well! (y/n) n

I'd expect, based on the docs that in both cases those external dependencies would be excluded? It's possible in the first instance that those dependencies would have been excluded, but the output makes me think they wouldn't be.. (excluded: false)

robarthur commented 3 years ago

I tried an explicit include as well... and seeing similar behaviour:

e.g.

$ terragrunt run-all destroy --terragrunt-include-dir ${PWD}
Module /aws/customer-account/transit_gateway_attachment depends on module /aws/networking-account/transit_gateway, which is an external dependency outside of the current working directory. Should Terragrunt run this external dependency? Warning, if you say 'yes', Terragrunt will make changes in /aws/networking-account/transit_gateway as well! (y/n) n

Module /aws/networking-account/transit_gateway depends on module /aws/networking-account/vpc, which is an external dependency outside of the current working directory. Should Terragrunt run this external dependency? Warning, if you say 'yes', Terragrunt will make changes in /aws/networking-account/vpc as well! (y/n) n

Edit:

Seeing the same behaviour with --terragrunt-strict-include

brikis98 commented 3 years ago

Hm, that's odd. --terragrunt-ignore-external-dependencies should do what you want...

Could you turn up the log level and share the full log output?

robarthur commented 3 years ago

@brikis98 Thanks for getting back. I turned the log level down to debug, see below. I redacted slightly less info this time, but hopefully it's still clear/relevant to the example above:

ubuntu@iboy:~/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application$ terragrunt run-all destroy --terragrunt-ignore-external-dependencies --terragrunt-log-level DEBUG
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, aws_account_id, aws_profile  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Evaluated 3 locals (remaining 3): region_vars, environment_vars, account_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, account_id, aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment] 
DEBU[0000] Setting download directory for module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment to /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment/.terragrunt-cache 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Evaluated 3 locals (remaining 0): aws_account_id, aws_profile, account_name  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 3 locals (remaining 0): region_vars, account_vars, environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Evaluated 3 locals (remaining 0): aws_account_id, aws_profile, account_name  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1] 
DEBU[0000] Evaluated 3 locals (remaining 3): environment_vars, account_vars, region_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, account_id, aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc] 
DEBU[0000] Setting download directory for module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc to /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc/.terragrunt-cache 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, aws_account_id, aws_profile  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1] 
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 3 locals (remaining 3): account_vars, region_vars, environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway] 
DEBU[0000] Evaluated 3 locals (remaining 0): aws_region, account_name, account_id  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway] 
DEBU[0000] Setting download directory for module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway to /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway/.terragrunt-cache 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, aws_account_id, aws_profile  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1] 
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1] 
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 1 locals (remaining 0): environment  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress] 
DEBU[0000] Evaluated 3 locals (remaining 3): account_vars, region_vars, environment_vars  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc] 
DEBU[0000] Evaluated 3 locals (remaining 0): account_name, account_id, aws_region  prefix=[/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc] 
DEBU[0000] Setting download directory for module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc to /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc/.terragrunt-cache 
INFO[0000] Stack at /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application:
  => Module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway (excluded: false, dependencies: [/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc])
  => Module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/vpc (excluded: false, dependencies: [])
  => Module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/transit_gateway_attachment (excluded: false, dependencies: [/home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/networking/us-east-1/egress/transit_gateway, /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc])
  => Module /home/ubuntu/Code/lib/infrastructure-as-code/terraform/aws/py3testa/us-east-1/application/vpc (excluded: false, dependencies: [])
robarthur commented 3 years ago

Hey @brikis98 I was wondering if there's was any more you could glean from he output above or anything you'd recommend trying?

brikis98 commented 3 years ago

Hm, that does seem like a bug. Unfortunately, I can't see from the log output what the cause is. We're buried at the moment, so if anyone has time to dig into this, a PR is very welcome.