search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
7.93k stars 964 forks source link

IMDSv1 errors when running terragrunt render-json in an EC2 environment without valid credentials #3308

Open jessebye opened 1 month ago

jessebye commented 1 month ago

Describe the bug

When running terragrunt render-json, we receive the following error over and over (multiple times per second until the operation is forcefully stopped):

SDK 2024/07/31 20:48:00 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed

It seems like this is happening because at this point in our CI pipeline, we haven't assumed an AWS role yet, so it can't find the credentials. However, this was working fine in an older Terragrunt version (0.50.6). Maybe the AWS SDK was updated and caused the behavior change?

For context, we use terragrunt render-json to parse our repo's HCL files and provide us with the AWS region and account ID that we need for the given resource. Then in a subsequent step we assume the correct AWS role before proceeding with plans/applies.

Steps To Reproduce

Run terragrunt render-json in an EC2 environment that has no valid AWS credentials.

Expected behavior

The command completes successfully, even when there are no valid AWS credentials.

Nice to haves

Versions

Additional context

n/a

denis256 commented 1 month ago

Hi, will be helpful to share output of execution with --terragrunt-debug to understand where it is failing

jessebye commented 1 month ago

@denis256 tried to do that, but because we have to interrupt the terragrunt command it doesn't seem to generate any debug output file:

2024-08-01T17:21:16.3341976Z ##[group]Run terragrunt render-json --terragrunt-debug
2024-08-01T17:21:16.3342644Z terragrunt render-json --terragrunt-debug
2024-08-01T17:21:16.3343427Z echo "aws_region=$(jq -r '.inputs.aws_region' terragrunt_rendered.json)" >> "$GITHUB_ENV"
2024-08-01T17:21:16.3344426Z echo "aws_account_id=$(jq -r '.inputs.aws_account_id' terragrunt_rendered.json)" >> "$GITHUB_ENV"
2024-08-01T17:21:16.3402528Z shell: /usr/bin/bash -e ***0***
2024-08-01T17:21:16.3402959Z env:
2024-08-01T17:21:16.3403269Z   tofu_version: 1.8.0
2024-08-01T17:21:16.3403632Z   tg_version: 0.64.4
2024-08-01T17:21:16.3406487Z   GIT_SSH_COMMAND: echo '***' > id_rsa && ssh-keyscan github.com > known_hosts && chmod 600 id_rsa known_hosts && ssh -i ./id_rsa -o UserKnownHostsFile=./known_hosts
2024-08-01T17:21:16.3407599Z ##[endgroup]
2024-08-01T17:21:17.0963278Z SDK 2024/08/01 17:21:17 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
2024-08-01T17:21:17.0998769Z SDK 2024/08/01 17:21:17 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
2024-08-01T17:21:17.1114190Z SDK 2024/08/01 17:21:17 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
[ repeat of the above about 1000x ]
2024-08-01T17:22:25.1591232Z ##[error]The operation was canceled.
2024-08-01T17:22:25.1771239Z ##[group]Run cat terragrunt-debug.tfvars
2024-08-01T17:22:25.1772072Z cat terragrunt-debug.tfvars
2024-08-01T17:22:25.1843543Z shell: /usr/bin/bash -e ***0***
2024-08-01T17:22:25.1844187Z env:
2024-08-01T17:22:25.1844648Z   tofu_version: 1.8.0
2024-08-01T17:22:25.1845193Z   tg_version: 0.64.4
2024-08-01T17:22:25.1849998Z   GIT_SSH_COMMAND: echo '***' > id_rsa && ssh-keyscan github.com > known_hosts && chmod 600 id_rsa known_hosts && ssh -i ./id_rsa -o UserKnownHostsFile=./known_hosts
2024-08-01T17:22:25.1851511Z ##[endgroup]
2024-08-01T17:22:25.1976924Z cat: terragrunt-debug.tfvars: No such file or directory
2024-08-01T17:22:25.1980403Z ##[error]Process completed with exit code 1.
jessebye commented 1 month ago

@denis256 ok, so I dialed up the log-level to trace and that gave some better info:

  terragrunt render-json --terragrunt-log-level trace
  echo "aws_region=$(jq -r '.inputs.aws_region' terragrunt_rendered.json)" >> "$GITHUB_ENV"
  echo "aws_account_id=$(jq -r '.inputs.aws_account_id' terragrunt_rendered.json)" >> "$GITHUB_ENV"
  shell: /usr/bin/bash -e ***0***
  env:
    tofu_version: 1.8.0
    tg_version: 0.64.4
    GIT_SSH_COMMAND: echo '***' > id_rsa && ssh-keyscan github.com > known_hosts && chmod 600 id_rsa known_hosts && ssh -i ./id_rsa -o UserKnownHostsFile=./known_hosts
time=2024-08-01T17:44:49Z level=debug msg=Terragrunt Version: 0.64.4
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/dev/backend.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/dev/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=Running command: tofu --version prefix=[/home/runner/work/infra-live/infra-live/dev/data-stores/aurora/shared] 
time=2024-08-01T17:44:49Z level=debug msg=tofu version: 1.8.0
time=2024-08-01T17:44:49Z level=debug msg=Reading Terragrunt config file at /home/runner/work/infra-live/infra-live/dev/data-stores/aurora/shared/terragrunt.hcl
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/dev/backend.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/dev/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Included config /home/runner/work/infra-live/infra-live/dev/backend.hcl has strategy shallow merge: merging config in (shallow) for dependency.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Included config /home/runner/work/infra-live/infra-live/dev/terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Included config /home/runner/work/infra-live/infra-live/dev/backend.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation.
time=2024-08-01T17:44:49Z level=debug msg=Included config /home/runner/work/infra-live/infra-live/dev/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
time=2024-08-01T17:44:49Z level=debug msg=git show-toplevel result: 

time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/dev] 
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/dev/data-stores/aurora/dataplatform-skeleton] 
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/dev/data-stores/aurora/dataplatform-skeleton] 
time=2024-08-01T17:44:49Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/dev/backend.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/dev/data-stores/aurora/dataplatform-skeleton] 
time=2024-08-01T17:44:49Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/dev/data-stores/aurora/dataplatform-skeleton] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/backend.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc3-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc3-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc3-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Found locals block: evaluating the expressions. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
SDK 2024/08/01 17:44:50 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
time=2024-08-01T17:44:50Z level=debug msg=Evaluated 1 locals (remaining 0): secret_vars prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/backend.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-migrations-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-migrations-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/backend.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-migrations-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-migrations-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=[Partial] Included config /home/runner/work/infra-live/infra-live/prod/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-cloud-node/cc4-migrations-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Found locals block: evaluating the expressions. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-indexer/cc4-database-user] 
SDK 2024/08/01 17:44:50 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
SDK 2024/08/01 17:44:50 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 403, request to EC2 IMDS failed
time=2024-08-01T17:44:50Z level=debug msg=Evaluated 1 locals (remaining 0): secret_vars prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-indexer/cc4-database-user] 
time=2024-08-01T17:44:50Z level=debug msg=Did not find any locals block: skipping evaluation. prefix=[/home/runner/work/infra-live/infra-live/prod/us-east-2/services/compliance-indexer/cc4-database-user]