New feature skip=true doesn't work

[aperigault]

Hello,

New feature skip=true introduced here to ignore root terraform.tfvars during xxx-all commands seems to not work.

I have a required_var_files in my terraform.tfvars :

`terragrunt = { skip = true

terraform { extra_arguments "-var-file" { commands = ["${get_terraform_commands_that_need_vars()}"]

  required_var_files = [
    "${get_tfvars_dir()}/${find_in_parent_folders("file1.tfvars")}",
    "${get_tfvars_dir()}/${find_in_parent_folders("file2.tfvars")}",
  ]
}

extra_arguments "custom_vars" {
  commands = ["${get_terraform_commands_that_need_vars()}"]

  arguments = [
    "-var", "remote_state_region=us-west-1",
    "-var", "remote_state_bucket=terraform-${get_aws_account_id()}",
  ]
}

} }`

but this file does not exist at the root level. So I attempt to skip this file but terragrunt seems to continue to read my file :

$ TERRAGRUNT_DEBUG=TRUE AWS_SDK_LOAD_CONFIG=1 AWS_PROFILE=TEST terragrunt apply-all [terragrunt] [test] 2019/05/14 12:23:53 Running command: terraform --version [terragrunt] 2019/05/14 12:23:55 configstack.ErrorProcessingModule Error processing module at '/test/terraform.tfvars'. How this module was found: Terragrunt config file found in a subdirectory of /test. Underlying error: Could not find a file2.tfvars in any of the parent folders of /test/terraform.tfvars. Cause: Traversed all the way to the root. /go/src/github.com/gruntwork-io/terragrunt/configstack/module.go:239 (0x9856e7) /go/src/github.com/gruntwork-io/terragrunt/configstack/module.go:215 (0x984e1a) /go/src/github.com/gruntwork-io/terragrunt/configstack/module.go:49 (0x98396a) /go/src/github.com/gruntwork-io/terragrunt/configstack/stack.go:128 (0x98aecf) /go/src/github.com/gruntwork-io/terragrunt/configstack/stack.go:110 (0x98ab7c) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:733 (0x99309f) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:681 (0x992a7e) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:228 (0x98f5d4) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:211 (0x98f2ce) /go/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:502 (0x92cc12) /go/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:268 (0x92aae3) /go/src/github.com/gruntwork-io/terragrunt/main.go:20 (0x997664) /usr/local/go/src/runtime/proc.go:195 (0x42bdf6) /usr/local/go/src/runtime/asm_amd64.s:2337 (0x458d11)

[brikis98]

The error is:

Could not find a file2.tfvars in any of the parent folders of /test/terraform.tfvars

Does that file exist? If not, then isn't required_var_files operating as expected?

[barryib]

@brikis98 those files exist, but not in parent folder of the root terraform.tfvars.

Actually, the repository structure is like this :

├── terraform.tfvars
└── region
    ├── prod
    │   ├── file1.tfvars
    │   ├── file2.tfvars
    │   ├── application_1
    │   │   └── terraform.tfvars
    │   ├── application_2
    │   │   └── terraform.tfvars
    │   ├── application_3
    │   │   └── terraform.tfvars

When the working directory is application_1 per exemple, we can run terragrunt without problem (it finds file1.tfvars and file2.tfvars with find_in_parent_folders()).

The root terraform.tfvars is only for DRY purpose. It define buckets and variables, but it doesn't define any infrastructure.

The problem is that, terragrunt is trying to evaluate interpolations (here find_in_parent_folders()) even if skip is set to true.

  required_var_files = [
    "${get_tfvars_dir()}/${find_in_parent_folders("file1.tfvars")}",
    "${get_tfvars_dir()}/${find_in_parent_folders("file2.tfvars")}",
  ]

Does that file exist? If not, then isn't required_var_files operating as expected?

Yes, those files are required, but not in the root level.

I think, with the skip flag, we should totally skip the terraform.tfvars it self (or it's evaluation in the current context).

[brikis98]

Oh, I gotcha, thanks for the explanation. Right now, the processing of interpolations such as find_in_parent_folders happens hand-in-hand with reading the config file itself, so the find_in_parent_folders function returns an error before we've even gotten to the skip check.

PR to improve this behavior is welcome!

[barryib]

@brikis98 I'll be happy to help on this. But I was wondering if you can point me the best place to start.

Should I skip interpolations here ? If yes, this won't break merge and includes ?

[brikis98]

@barryib Thx for the offer. You may want to hold off a bit until https://github.com/gruntwork-io/terragrunt/issues/466 is resolved, as that will change interpolation processing quite a bit.

[aperigault]

Hi, now it's time to fix this really annoying issue. @brikis98 is @barryib in the right way ?

[dudicoco]

Hi,

I'm experiencing a similar issue with skip = true in the child terragrunt.hcl files not working due to missing yaml files in the directory.

The folloeinh is defined in the root inputs block:

inputs = merge(
  yamldecode(
    file("${get_parent_terragrunt_dir()}/../global/global-values_${get_env("ENVIRONMENT", "")}.yaml"),
  ),
  yamldecode(
    file("${get_terragrunt_dir()}/values_${get_env("ENVIRONMENT", "")}.yaml"),
  )
)

Any ideas how to solve this? Thanks

[brikis98]

Use fileexists and a conditional (COND ? TRUEVAL : FALSEVAL) to only read the file in if it exists?

[dudicoco]

Thanks @brikis98. I've already done something similar in the child terragrunt.hcl file: https://github.com/gruntwork-io/terragrunt/issues/989#issuecomment-569991611 I was also thinking of moving the fileexists conditional to the main terragrunt.hcl file, might give it a try.

[dudicoco]

Actually I think I would still need to fileexists conditional in the child terragrunt.hcl for the skip function.

[jacksonporter]

Has there been progress on this issue? This is a blocker on DRYing up my terragrunt configurations.

[xgt001]

can you please expand the current documentation around the skip = true functionality to mention the current gotchas of its conflicts with other interpolation functions ? Atleast per the documentation here it looks like skip is supposed to skip the module execution entirely unless the children include it

I have a config of something like this

skip = true
terraform {
  source = "../modules/vpc"
}

locals {
  region_mapping = {
    bar = {
      vpc_key = "a"
    },
    foo = {
      vpc_key = "a"
    }
    common = {
      vpc_key = "a"
    }
  }

  parsed_path = try(regex(".*/configs/(?P<account>.*?)/(?P<region>.*?)/(?P<module>.*)", abspath(".")), {
    account = "common"
    region  = "us-east-1"
    module  = "vpc-outputs"
  })
  account        = local.parsed_path.account
  region_name    = local.parsed_path.region
  module         = local.parsed_path.module
  account_config = try(read_terragrunt_config("${get_terragrunt_dir()}/../../account.hcl"), read_terragrunt_config("../../foo/account.hcl"))
  account_id     = local.account_config.locals.account_id
}

inputs = {
  vpc_key = local.region_mapping[local.account].vpc_key
}

#include the provider settings
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
    provider "aws" {
      region = "${local.region_name}"

    }
    EOF
}

Coupled with a folder structure of

.
├── terragrunt.hcl
└── configs/
    ├── common/
    │   └── vpc-outputs/
    │       └── terragrunt.hcl - only for DRY
    ├── account/
    │   ├── region/
    │   │   ├── cluster/
    │   │   │   └── terragrunt.hcl
    │   │   ├── vpc-outputs/
    │   │   │   └── terragrunt.hcl - I want this to execute
    │   │   └── terragrunt.hcl
    │   └── terragrunt.hcl
    └── terragrunt.hcl

Without the try safeguards, the hcl was still getting evaluated with run-all plan and breaking due to skip not working. Looks like from @brikis98 's remark https://github.com/gruntwork-io/terragrunt/issues/704#issuecomment-492618513 , it seems to be failing due to the directory functions I am invoking?

Also - please do let me know if I am trying something wrong :)

[GreasyAvocado]

Are there any plans to implement this (making skip=true actually skip the module altogether)?

[velkovb]

I am experiencing the same issue. Are there any plans on revisiting this?

At least an update in the documentation about this issue and possible workaround would be great.

[perplexa]

@velkovb I use the following as workaround:

find . -iname terragrunt.hcl -not -path '*/.*/*' -print0 | xargs -0 grep -L '^skip *= *true' | tr '\n' '\0' | xargs -0 -n 1 -I % -- sh -c 'cd "$(dirname "%")" && terragrunt plan'

While it is not ideal, at least it works.

[gbx-jzakrzeski]

Ran into this dealing with a couple conditionally-skipped modules.

My problem was that I have two modules that I only want to run when I'm planning/applying my production environment. Otherwise, these are ignored. The first skipped module creates a resource, and the second skipped module depends on outputs of the first. I've gotten around it by conditionally including "apply" for mock_outputs_allowed_terraform_commands in the dependency block in the second skipped module so that it would get far enough to evaluate that it actually just needs to skip the module.

e.g.:

skip = local.env == "prod" ? false : true

  ...

locals {
  environment_vars = read_terragrunt_config(find_in_parent_folders("${get_env("TF_VAR_env")}_variables.hcl"))
  env = local.environment_vars.locals.env
}

dependency "foo" {
  config_path = "some/path/to/another/skipped/module"

  mock_outputs = {
    host = "foo.bar.com"
  }

  mock_outputs_allowed_terraform_commands = local.env == "prod" ? ["validate", "plan", "providers", "init"] : ["validate", "plan", "providers", "init", "apply"]
}

inputs = {
  hostname = dependency.foo.outputs.host
}

The basic problem is that skip isn't checked until all the expressions are evaluated, which makes sense since you may need some of those expressions evaluated to set skip given its purpose. Not sure there's a good way to actually fix it unless you just trace the values for the skip directive, eval those eagerly, then do the rest if you need to.