search

gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
8.09k stars 981 forks source link

[bug] Crashing when run from an alpine image #991

Open slimm609 opened 4 years ago

slimm609 commented 4 years ago

We are running into a crash when run in a pipeline on an m5.xlarge instance in docker alpine. The terraform runs fine on a t3 instance through the same pipeline but not in m5 for some reason.

--- Stack trace ---

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xd02d21] goroutine 8 [running]: github.com/gruntwork-io/terragrunt/cli.runTerraformWithRetry(0xc0000c2000, 0x3, 0x0) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:487 +0x121 github.com/gruntwork-io/terragrunt/cli.runTerragruntWithConfig.func1(0xc0002d4510, 0x2) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:440 +0x2a github.com/gruntwork-io/terragrunt/cli.runActionWithHooks(0xf56322, 0x9, 0xc0000c2000, 0xc0002d4750, 0xc0001c9bf0, 0x0, 0x44) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:451 +0x2ae github.com/gruntwork-io/terragrunt/cli.runTerragruntWithConfig(0xc0000c2000, 0xc0002d4750, 0xc0000c2000, 0x0, 0x0) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:439 +0x2c7 github.com/gruntwork-io/terragrunt/cli.runTerraformInit(0xc0000c38c0, 0xc0002d4750, 0x0, 0x0, 0x0) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:632 +0xaa github.com/gruntwork-io/terragrunt/cli.prepareNonInitCommand(0xc0000c38c0, 0xc0002d4750, 0x0, 0x0) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:575 +0x8b github.com/gruntwork-io/terragrunt/cli.runTerragruntWithConfig(0xc0000c38c0, 0xc0002d4750, 0x0, 0x0, 0x0) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:430 +0x211 github.com/gruntwork-io/terragrunt/cli.RunTerragrunt(0xc0000c38c0, 0xf6082b, 0x15) /go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:327 +0x51b github.com/gruntwork-io/terragrunt/configstack.(runningModule).runNow(0xc000482500, 0x0, 0x0) /go/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:238 +0x17a github.com/gruntwork-io/terragrunt/configstack.(runningModule).runModuleWhenReady(0xc000482500) /go/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:201 +0x6a github.com/gruntwork-io/terragrunt/configstack.runModules.func1(0xc0003a3b80, 0xc000482500) /go/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:171 +0x51 created by github.com/gruntwork-io/terragrunt/configstack.runModules /go/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:169 +0xe1

brikis98 commented 4 years ago

Weird. Any other differences between the two? Different OS or same? What version of Terragrunt?

slimm609 commented 4 years ago

Same OS, both are amazon linux 2 with docker installed.

TF version is 0.12.18 TG version is 0.21.10

slimm609 commented 4 years ago

They are both running on amazon linux 2 images via gitlab CI and both the same gitlab-runner configurations. The only difference is the size/type of instances because 1 account has more jobs then the other so it is larger to handle higher concurrency, everything else is the same.

brikis98 commented 4 years ago

From the stack trace you posted, it seems to be hitting a nil pointer dereference here: https://github.com/gruntwork-io/terragrunt/blob/v0.21.10/cli/cli_app.go#L487

The only reference I see is out.Stderr, so that would imply out is nil. Depending on where the error happened in the RunShellCommandWithOutput method, it's definitely possible for it to return nil for the CmdOutput parameter. What's not clear is why it would return an error on one instance type but not another.

That said, it's worth updating the if-statement here to check out is not nil before trying to reference data in it. If we do that, it'll fix the panic, and might let us understand more about what the underlying error is. Do you have any interest in submitting a quick PR to tweak this if-statement?

slimm609 commented 4 years ago

I’ll take a look at it, my go code isn’t the greatest but I will look. I looked at the logs more and it’s happening in the terraform init.

slimm609 commented 4 years ago

Including the command that caused the panic

[terragrunt] [/builds/terraform/aws/us-west-2/bless] 2020/01/09 00:04:18 Setting working directory to /builds/terraform/aws/us-west-2/bless/.terragrunt-cache/UmMlsotVX2PpKdWF9uY3Ok/vQS_ZBbJ1UkQsnbYsPThpRs [terragrunt] [/builds/terraform/aws/us-west-2/bless] 2020/01/09 00:04:18 Initializing remote state for the s3 backend [terragrunt] [/builds/terraform/aws/us-west-2/bless] 2020/01/09 00:04:18 Detected 2 Hooks [terragrunt] [/builds/terraform/aws/us-west-2/bless] 2020/01/09 00:04:18 Running command: terraform init -backend-config=bucket=terraform-abcd -backend-config=dynamodb_table=terraform-lock -backend-config=encrypt=true -backend-config=key=us-west-2/bless/terraform.tfstate -backend-config=region=us-west-2 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xd02d21] goroutine 23 [running]: