Closed chris1248 closed 3 years ago
@chris1248 Could you provide more info on your Terragrunt configurations?
@yorinasub17 Could this be related to https://github.com/gruntwork-io/terragrunt/releases/tag/v0.28.3 (https://github.com/gruntwork-io/terragrunt/pull/1533)?
Yes this looks like it is related.
@chris1248 (assuming you have remote_state
configured) could you try wiping .terragrunt-cache
from the /bla bla bla
folder (the module targeted by dependency
) and see if the error message changes?
Separately, was this an existing config that you upgraded to tf 0.14, or completely brand new?
Just ran into this myself and it does appear to be due to an optimization, but not the one made in v0.28.3
and clearing the .terragrunt-cache
doesn't solve it (actually was quite the head scratcher at first because everything worked locally where each module was already initialized, but the error showed up in CI, and eventually I managed to reproduce locally by clearing the .terragrunt-cache
directories to test with the same clean state you'd have in a CI build)
Setting disable_dependency_optimization = true
seems to be an effective workaround, which runs succesuflly albeit with a couple warnings about falling back to using terragrunt output
which is likely why it works since in such a case the init in dependencies would be implied by running the output command.
WARN[0000] Could not parse remote_state block from target config /Users/davidalger/Work/algerdev/live/project/terragrunt.hcl
WARN[0000] Falling back to terragrunt output.
WARN[0000] Could not parse remote_state block from target config /Users/davidalger/Work/algerdev/live/cluster/terragrunt.hcl
WARN[0000] Falling back to terragrunt output.
The issue is reproducible all the way back to v0.23.38
(the only thing in this release was #1321). The issue cannot be reproduced in v0.23.37
. Here is what I was doing to test each version stepping back to where it stopped breaking:
version=v0.23.37
wget https://github.com/gruntwork-io/terragrunt/releases/download/$version/terragrunt_darwin_amd64
mv terragrunt_darwin_amd64 /tmp/terragrunt-$version
chmod +x /tmp/terragrunt-$version
find . -type d -name .terragrunt-cache -exec rm -rf {} +
(cd live/traefik; /tmp/terragrunt-$version init)
For now I'm going to set disable_dependency_optimization
in my config so it doesn't become a blocker for me.
My directory tree looks like this:
├── common_vars.hcl
├── live
│ ├── cluster
│ │ └── terragrunt.hcl
│ ├── project
│ │ └── terragrunt.hcl
│ └── traefik
│ └── terragrunt.hcl
└── terragrunt.hcl
Relevant Terragrunt config:
davidalger@silver-bullet:~/Work/algerdev (terragrunt *) $ cat live/traefik/terragrunt.hcl
terraform {
source = "../../modules//traefik"
}
include {
path = find_in_parent_folders()
}
dependency "project" {
config_path = "../project"
}
dependency "cluster" {
config_path = "../cluster"
}
inputs = {
managed_zone_name = dependency.project.outputs.managed_zone_name
managed_zone_dns_name = dependency.project.outputs.managed_zone_dns_name
}
# Generate providers required to deploy into GKE cluster
generate "provider_cluster" {
path = "provider_cluster.tf"
if_exists = "overwrite_terragrunt"
contents = <<-CONTENTS
# Provides OAuth token for K8S authentication
data "google_client_config" "default" {}
provider "kubernetes" {
host = "${dependency.cluster.outputs.gke.endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = <<-EOT
${indent(4, dependency.cluster.outputs.gke.cluster_ca_certificate)}
EOT
}
provider "kubernetes-alpha" {
host = "https://${dependency.cluster.outputs.gke.endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = <<-EOT
${indent(4, dependency.cluster.outputs.gke.cluster_ca_certificate)}
EOT
}
provider "helm" {
kubernetes {
host = "${dependency.cluster.outputs.gke.endpoint}"
token = data.google_client_config.default.access_token
cluster_ca_certificate = <<-EOT
${indent(6, dependency.cluster.outputs.gke.cluster_ca_certificate)}
EOT
}
}
CONTENTS
}
davidalger@silver-bullet:~/Work/algerdev (terragrunt *) $ cat terragrunt.hcl
locals {
common_vars = lookup(read_terragrunt_config(find_in_parent_folders("common_vars.hcl", "_"), {}), "locals", {})
}
# Generate Google provider block
generate "provider" {
path = "provider.tf"
if_exists = "overwrite_terragrunt"
contents = <<-EOT
terraform {
backend "gcs" {
bucket = "${local.common_vars.gcp_tfstate_bucket}"
prefix = "${path_relative_to_include()}"
}
}
provider "google" {
region = "${local.common_vars.gcp_region}"
project = "${local.common_vars.gcp_project}"
impersonate_service_account = data.terraform_remote_state.project.outputs.terraform_service_account
}
provider "google-beta" {
region = "${local.common_vars.gcp_region}"
project = "${local.common_vars.gcp_project}"
impersonate_service_account = data.terraform_remote_state.project.outputs.terraform_service_account
}
data "terraform_remote_state" "project" {
backend = "gcs"
config = {
bucket = "${local.common_vars.gcp_tfstate_bucket}"
prefix = "live/project"
}
}
EOT
}
# Configure Terragrunt to automatically store tfstate files in a GCS bucket
remote_state {
backend = "gcs"
config = {
bucket = local.common_vars.gcp_tfstate_bucket
prefix = path_relative_to_include()
skip_bucket_creation = true
skip_bucket_versioning = true
}
}
# Global parameters which apply to all configurations
inputs = {
region = local.common_vars.gcp_region,
project = local.common_vars.gcp_project,
network = local.common_vars.gcp_network,
}
# Nothing in here to apply
skip = true
And lastly here is a trace leading up the error from Terraform:
davidalger@silver-bullet:~/Work/algerdev (terragrunt *) $ find . -type d -name .terragrunt-cache -exec rm -rf {} +
davidalger@silver-bullet:~/Work/algerdev (terragrunt *) $ (cd live/traefik; terragrunt init --terragrunt-log-level debug)
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_project, gcp_network, gcp_region, gcp_tfstate_bucket prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars
DEBU[0000] Running command: terraform --version prefix=[/Users/davidalger/Work/algerdev/live/traefik]
DEBU[0000] Terraform version: 0.15.0
DEBU[0000] Reading Terragrunt config file at /Users/davidalger/Work/algerdev/live/traefik/terragrunt.hcl
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_network, gcp_region, gcp_tfstate_bucket, gcp_project prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_project, gcp_network, gcp_region, gcp_tfstate_bucket prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Getting output of dependency /Users/davidalger/Work/algerdev/live/cluster/terragrunt.hcl for config /Users/davidalger/Work/algerdev/live/traefik/terragrunt.hcl
DEBU[0000] Getting output of dependency /Users/davidalger/Work/algerdev/live/project/terragrunt.hcl for config /Users/davidalger/Work/algerdev/live/traefik/terragrunt.hcl
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_project, gcp_network, gcp_region, gcp_tfstate_bucket prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_region, gcp_tfstate_bucket, gcp_project, gcp_network prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Did not find any locals block: skipping evaluation. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_project, gcp_network, gcp_region, gcp_tfstate_bucket prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 4 locals (remaining 0): gcp_project, gcp_network, gcp_region, gcp_tfstate_bucket prefix=[/Users/davidalger/Work/algerdev]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Evaluated 1 locals (remaining 0): common_vars prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Detected remote state block with generate config. Resolving dependency by pulling remote state. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Detected remote state block with generate config. Resolving dependency by pulling remote state. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Setting dependency working directory to /Users/davidalger/Work/algerdev/live/cluster/.terragrunt-cache/796666379 prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Setting dependency working directory to /Users/davidalger/Work/algerdev/live/project/.terragrunt-cache/800269806 prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Generated file /Users/davidalger/Work/algerdev/live/cluster/.terragrunt-cache/796666379/backend.tf. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Generated remote state configuration in working dir /Users/davidalger/Work/algerdev/live/cluster/.terragrunt-cache/796666379 prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Running command: terraform init -get=false -get-plugins=false prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0000] Generated file /Users/davidalger/Work/algerdev/live/project/.terragrunt-cache/800269806/backend.tf. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Generated remote state configuration in working dir /Users/davidalger/Work/algerdev/live/project/.terragrunt-cache/800269806 prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0000] Running command: terraform init -get=false -get-plugins=false prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0001] Ignoring expected error from dependency init call prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0001] Init call stderr: prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0001] Usage: terraform [global options] init [options]
Initialize a new or existing Terraform working directory by creating
initial files, loading any remote state, downloading modules, etc.
This is the first command that should be run for any new or existing
Terraform configuration per machine. This sets up all the local data
necessary to run Terraform that is typically not committed to version
control.
This command is always safe to run multiple times. Though subsequent runs
may give errors, this command will never delete your configuration or
state. Even so, if you have important information, please back it up prior
to running this command, just in case.
Options:
-backend=true Configure the backend for this configuration.
-backend-config=path This can be either a path to an HCL file with key/value
assignments (same format as terraform.tfvars) or a
'key=value' format. This is merged with what is in the
configuration file. This can be specified multiple
times. The backend type must be in the configuration
itself.
-force-copy Suppress prompts about copying state data. This is
equivalent to providing a "yes" to all confirmation
prompts.
-from-module=SOURCE Copy the contents of the given module into the target
directory before initialization.
-get=true Download any modules for this configuration.
-input=true Ask for input if necessary. If false, will error if
input was required.
-no-color If specified, output won't contain any color.
-plugin-dir Directory containing plugin binaries. This overrides all
default search paths for plugins, and prevents the
automatic installation of plugins. This flag can be used
multiple times.
-reconfigure Reconfigure the backend, ignoring any saved
configuration.
-upgrade=false If installing modules (-get) or plugins, ignore
previously-downloaded objects and install the
latest version allowed within configured constraints.
-lockfile=MODE Set a dependency lockfile mode.
Currently only "readonly" is valid. prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0001] Running command: terraform output -json prefix=[/Users/davidalger/Work/algerdev/live/project]
DEBU[0001] Ignoring expected error from dependency init call prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0001] Init call stderr: prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0001] Usage: terraform [global options] init [options]
Initialize a new or existing Terraform working directory by creating
initial files, loading any remote state, downloading modules, etc.
This is the first command that should be run for any new or existing
Terraform configuration per machine. This sets up all the local data
necessary to run Terraform that is typically not committed to version
control.
This command is always safe to run multiple times. Though subsequent runs
may give errors, this command will never delete your configuration or
state. Even so, if you have important information, please back it up prior
to running this command, just in case.
Options:
-backend=true Configure the backend for this configuration.
-backend-config=path This can be either a path to an HCL file with key/value
assignments (same format as terraform.tfvars) or a
'key=value' format. This is merged with what is in the
configuration file. This can be specified multiple
times. The backend type must be in the configuration
itself.
-force-copy Suppress prompts about copying state data. This is
equivalent to providing a "yes" to all confirmation
prompts.
-from-module=SOURCE Copy the contents of the given module into the target
directory before initialization.
-get=true Download any modules for this configuration.
-input=true Ask for input if necessary. If false, will error if
input was required.
-no-color If specified, output won't contain any color.
-plugin-dir Directory containing plugin binaries. This overrides all
default search paths for plugins, and prevents the
automatic installation of plugins. This flag can be used
multiple times.
-reconfigure Reconfigure the backend, ignoring any saved
configuration.
-upgrade=false If installing modules (-get) or plugins, ignore
previously-downloaded objects and install the
latest version allowed within configured constraints.
-lockfile=MODE Set a dependency lockfile mode.
Currently only "readonly" is valid. prefix=[/Users/davidalger/Work/algerdev/live/cluster]
DEBU[0001] Running command: terraform output -json prefix=[/Users/davidalger/Work/algerdev/live/cluster]
╷
│ Error: Initialization required. Please see the error message above.
│
│
╵
╷
│ Error: Initialization required. Please see the error message above.
│
│
╵
ERRO[0001] exit status 1
I always wipe the terragrunt cache directory but it doesn't matter. I solved this by reverting to an older version of terragrunt. Sorry guys, new stuff has wierd bugs, we don't use it.
@yorinasub17 @brikis98 I think what I'm finding points to possibly needing to simply revert the optimization from #1321, at the very least remove the use of -get-plugins=false
which is what's causing the error. As can be seen from my trace, the terraform init
call is printing a help message, not an error or warning which is being ignored as an "expected error from dependency init call" despite the error being that an invalid flag is being passed.
From the changelog for Terraform v014.3:
terraform init: setting -get-plugins to false will now cause a warning, as this flag has been a no-op since 0.13.0 and usage is better served through using provider_installation blocks (#27092)
From the changelog for Terraform v0.15.0-beta1:
The -get-plugins=false option is no longer available on terraform init. (Terraform now always installs plugins.) (#27463)
With -get-plugins
being a no-op since 0.13.0, it's a bit befuddling why it was added as an optimization circa August 2020, but it was, oh well, and now it's breaking things since the flag is straight up gone. This does lead me to make an additional observation worth mentioning: Upon further inspection of the trace from the OP here, the original issue encountered may be different than the one in my trace as I was running 0.15.0-beta2
when I encountered it, and @chris1248 indicated it broke with 0.14.7 where the flag, although it doesn't do anything, still actually exists and shouldn't trigger the issue shown in my similar but different trace.
I haven't attempted running tests, but building Terragrunt to include the changes in https://github.com/davidalger/terragrunt/commit/cb125214a97481a51170461965d633e9d1f58809 resolves the issue I was having at least (doing more testing, and going to try and run the test suite before submitting a PR for this one). Given my further discovery efforts, I doubt it resolves the OPs, and that issue may indeed be related to the work in #1533 idk.
Please give https://github.com/gruntwork-io/terragrunt/releases/tag/v0.28.17 a shot (new binaries should show up in the next few minutes). Thanks to @davidalger's PR, I believe that should fix the issue.
I have tested out version 0.28.17 with three different versions of terraform: 0.12.28 - it works 0.14.7 - it works 0.14.9 - it works
I am trying to run terragrunt plan, but when I do I get a very unhelpful error:
Error: Initialization required. Please see the error message above.
The full command:
terragrunt plan --terragrunt-log-level debug --terragrunt-debug --terragrunt-working-dir working_dir
And the full output: