Terragrunt dependencies are not included since update v0.54.16

applevladko commented 1 month ago

Summary

Hi everyone! We are using terragrunt on main module and dependencies from there were included to the sub-folders of this module. Both of them are include general configuration of our terragrunt setup (e,g terragrunt/terragrunt.hcl )

Motivation

Upgrade terragrunt to latest version.

Proposal

I can see a lot of dependencies issues since this version upgrade but none of exact my cases. Maybe right way to do this should be added or documentaion or, if it no possible, this is regression.

Technical Details

Example:

sub-folder

include {
  path = find_in_parent_folders("terragrunt/terragrunt.hcl")
}

inputs = read_terragrunt_config("..").inputs

main folder

include {
  path = find_in_parent_folders("terragrunt/terragrunt.hcl")
}

dependency "s3" {
  config_path                            = "../s3"
  mock_outputs_merge_strategy_with_state = local.mock_outputs_merge_strategy_with_state
  mock_outputs = {
    example-s3        = "mock-example-s3"
    example2-s3        = "mock-example2-s3"
    example3-s3        = "mock-example3-s3"
...
    }
  }
}

locals {
  inputs = {
    team    = "devops"
  }

inputs = merge(local.inputs, local.env_inputs[local.env], {
  example-s3             = dependency.s3.outputs.example-s3 
  example2-s3           = dependency.s3.outputs.example2-s3
  example3-s3           = dependency.s3.outputs.example3-s3
...
})

dependencies {
  paths = ["../s3"]
}

But after config-read changes in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.54.16 we are unfortunately unable to do so, I've tried to rewrite configs but dependencies inputs are marked as not found in the sub-module (in main module works well so I believe inputs = read_terragrunt_config("..").inputs is not working)

15:26:31.022 ERROR  Error: Unsupported attribute

15:26:31.022 ERROR    on example/sub-folder/terragrunt.hcl line 67:

15:26:31.023 ERROR    67:   example-s3= dependency.s3.outputs.example-s3

15:26:31.023 ERROR  

15:26:31.023 ERROR  This object does not have an attribute named "s3".

So cause of thie issues we had to freeze out version on the "<= 0.54.15" yet. @levkohimins Maybe you could help with the new config read method?

Thanks in advance!

Press Release

none

Drawbacks

No response

Alternatives

No response

Migration Strategy

No response

Unresolved Questions

No response

References

No response

Proof of Concept Pull Request

No response

Support Level

[ ] I have Terragrunt Enterprise Support
[ ] I am a paying Gruntwork customer

Customer Name

No response

denis256 commented 1 month ago

Hi, this is a rfc or bug?

applevladko commented 1 month ago

Hello!

We've encountered an issue with our workflow since tried to update to the specified version. I'm not entirely sure if it's a bug or if I'm misunderstanding how to properly implement this in the newer version. Any guidance would be appreciated!

inetshell commented 1 month ago

I can confirm that this problem started with version v0.54.16:

With version v0.54.15, this code works fine:

dependency "vpc" {
  config_path = find_in_parent_folders("vpc")
}

dependencies {
  paths = [
    find_in_parent_folders("vpc")
  ]
}

locals {
  # Import variables
  org_settings     = read_terragrunt_config(find_in_parent_folders("org_settings.hcl")).inputs
  project_settings = read_terragrunt_config(find_in_parent_folders("_configuration/project.hcl")).inputs
  client           = local.project_settings.client
  project_id       = local.project_settings.project_id
  region           = local.project_settings.region
  zone             = local.project_settings.zone
  zones            = local.project_settings.zones

  # Module variables
}

terraform {
  source = "tfr:///HobOps/project-template/google//.?version=0.4.0"
}

inputs = {
    "ip-internal-loadbalancer-${local.region}" = {
      region       = local.region
      address_type = "INTERNAL"
      subnetwork   = dependency.vpc.outputs.vpc["vpc-main"].subnets["${local.region}/subnet-management-${local.region}"].name
    }
}

With version v0.54.16, this code works fine (I'm using the subnetwork name directly):

dependency "vpc" {
  config_path = find_in_parent_folders("vpc")
}

dependencies {
  paths = [
    find_in_parent_folders("vpc")
  ]
}

locals {
  # Import variables
  org_settings     = read_terragrunt_config(find_in_parent_folders("org_settings.hcl")).inputs
  project_settings = read_terragrunt_config(find_in_parent_folders("_configuration/project.hcl")).inputs
  client           = local.project_settings.client
  project_id       = local.project_settings.project_id
  region           = local.project_settings.region
  zone             = local.project_settings.zone
  zones            = local.project_settings.zones

  # Module variables
}

terraform {
  source = "tfr:///HobOps/project-template/google//.?version=0.4.0"
}

inputs = {
    "ip-internal-loadbalancer-${local.region}" = {
      region       = local.region
      address_type = "INTERNAL"
      // subnetwork   = dependency.vpc.outputs.vpc["vpc-main"].subnets["${local.region}/subnet-management-${local.region}"].name
      subnetwork   = "${local.region}/subnet-management-${local.region}"
    }
}

But with version v.0.54.16, when trying do use the dependency.vpc.outputs.vpc with this code:

dependency "vpc" {
  config_path = find_in_parent_folders("vpc")
}

dependencies {
  paths = [
    find_in_parent_folders("vpc")
  ]
}

locals {
  # Import variables
  org_settings     = read_terragrunt_config(find_in_parent_folders("org_settings.hcl")).inputs
  project_settings = read_terragrunt_config(find_in_parent_folders("_configuration/project.hcl")).inputs
  client           = local.project_settings.client
  project_id       = local.project_settings.project_id
  region           = local.project_settings.region
  zone             = local.project_settings.zone
  zones            = local.project_settings.zones

  # Module variables
}

terraform {
  source = "tfr:///HobOps/project-template/google//.?version=0.4.0"
}

inputs = {
    "ip-internal-loadbalancer-${local.region}" = {
      region       = local.region
      address_type = "INTERNAL"
      subnetwork   = dependency.vpc.outputs.vpc["vpc-main"].subnets["${local.region}/subnet-management-${local.region}"].name
      // subnetwork   = "${local.region}/subnet-management-${local.region}"
    }
}

I got this error:

ERRO[0000] /home/xxx/network-addresses/terragrunt.hcl:21,3-26: Error in function call; Call to function "read_terragrunt_config" failed: /home/xxx/_configuration/network-addresses.hcl:32,32-36: Unsupported attribute; This object does not have an attribute named "vpc".., and 4 other diagnostic(s) 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

My workaround was to use version v0.54.15.

inetshell commented 1 month ago

Hi, this is a rfc or bug?

It's a bug @denis256 , I can confirm that multiple instances of the same code stopped working after version v0.54.16. After comparing both versions, I saw a lot of references to dependencies being modified: https://github.com/gruntwork-io/terragrunt/compare/v0.54.15...v0.54.16

yhakbar commented 1 month ago

Hey folks, can we have a simplified example that we can use to reproduce the issue?

I'm having trouble understanding what the issue is, exactly.

inetshell commented 1 month ago

I'm still trying to generate a simplified example code, but when doing it, the code works on both versions. @applevladko can you help us reproducing the problem?

applevladko commented 1 month ago

Hello! Sorry for late response. Here is the sample setup which could demonstrate the issue:

we have file infra/terragrunt.hcl that includes to all ours terragrunt files with default (tags backend state region etc)

locals {
  env        = lower(get_env("ENV"))
  aws_region = "eu-central-1"

  repo_subpath = replace(replace("${path_relative_to_include()}/default.tfstate", "../", ""), "envs/", "")
}

inputs = {
  env        = local.env
  aws_region = local.aws_region
}

remote_state {
  backend = "s3"
  config = {
    bucket = "tf-state.${local.env}.test"
    region = local.aws_region

    encrypt                  = false
    skip_bucket_enforced_tls = true
    skip_bucket_ssencryption = true
    skip_bucket_root_access  = true
  }
}

generate "default_vars" {
  path      = "_variables.tf"
  if_exists = "overwrite_terragrunt"
  contents  = file("variables.tf")
}

generate "provider" {
  path      = "_providers.tf"
  if_exists = "overwrite_terragrunt"
  contents  = file("providers.tf")
}

we have folder test-bucket that creates bucket and set it the outputs

test-bucket/bucket.tf file

module "s3_test_tg_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "4.2.0"

  bucket = "test-terragrunt-bucket2024"

  control_object_ownership = true
}
locals {
  buckets = {
    test = module.s3_test_tg_bucket
  }
}

output "test-bucket" {
  description = "The name of the raw bucket"
  value       = module.s3_test_tg_bucket.s3_bucket_id
}

output "test-bucket-arn" {
  description = "The arn of the raw bucket"
  value       = module.s3_test_tg_bucket.s3_bucket_arn
}

test-bucket/terragrunt.hcl file

terraform_version_constraint  = ">= 1.3"
terragrunt_version_constraint = "<= 0.54.15"

include {
  path = find_in_parent_folders("infra/terragrunt.hcl")
}

locals {
  env = get_env("ENV")

  mock_outputs_merge_strategy_with_state = get_env("TF_MOCK_STRATEGY", "no_merge")

  inputs = {
    team    = "test"
  }
}

inputs = local.inputs

other module containes dummy files with dependency and included submodule in subfolder:

test-module/test-module.tf

terraform {
  required_version = ">= 1.3.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.29.0"
    }
  }
}

test-module/terragrunt.hcl

terraform_version_constraint  = ">= 1.3"
terragrunt_version_constraint = ">= 0.54.15"

include {
  path = find_in_parent_folders("infra/terragrunt.hcl")
}

dependency "test-bucket" {
  config_path                            = "../test-bucket"
  mock_outputs_merge_strategy_with_state = local.mock_outputs_merge_strategy_with_state
  mock_outputs = {
    test-bucket        = "mock-test-bucket"
  }
}

locals {
  env = get_env("ENV")

  mock_outputs_merge_strategy_with_state = get_env("TF_MOCK_STRATEGY", "no_merge")

  inputs = {
    team    = "test"
  }
}

inputs = merge(local.inputs, {
  test-bucket             = dependency.test-bucket.outputs.test-bucket
})

dependencies {
  paths = ["../test-bucket"]
}

test-module/test-included-module/terragrunt.hcl

terraform_version_constraint  = ">= 1.3"
terragrunt_version_constraint = "<= 0.54.15"

include {
  path = find_in_parent_folders("infra/terragrunt.hcl")
}

inputs = read_terragrunt_config("..").inputs

thats the whole minimal setup. terragrunt init will not work in folder test-module/test-included-module/ for version newer then 0.54.15 as the dependencies from upper module will not be found. while it works in <= 0.54.15 so the issue is that the part withinputs = read_terragrunt_config("..").inputs does not work for dependencies anymore.

gruntwork-io / terragrunt