Closed internetti closed 11 months ago
jbardin
commented
11 months ago Hi @internetti,
Can you verify that you have upgraded to Terraform v1.6.1? This stack trace you've shown here is from v1.6.0 and should be fixed in the current release.
Thanks!
micaroni
commented
11 months ago I've been having this same issue. Like internetti, I am running v1.6.1. My state file shows "terraform_version": "1.5.7", which I believe from reading through other issues, should also not cause this error (though I could be mistaken).
This only started happening for me after attempting to add a large number of imported resources, but even then, not immediately.
jbardin
commented
11 months ago The stack trace here cannot come from v1.6.1, and is probably from v1.6.0. Make sure you're running terraform version from the same location where Terraform is executing. If Terraform is running remotely in TFC, make sure the environment there is also showing the correct version. The state file will only show which version was last to change that state file, so should be no factor here.
micaroni
commented
11 months ago Yep, running terraform version from the same location where Terraform is executing. I am currently only running Terraform locally, not remotely in TFC (or anywhere else).
What else can be done to troubleshoot this? I've made sure the provider is also up-to-date.
jbardin
commented
11 months ago I'm not sure if the complete trace logs would help, since this seems to be the entirely wrong version which is running. Maybe just capture the continuous output from TF_LOG_CORE=info terraform plan to verify what is happening. That should have lines near the start showing the version running and the cli arguments, and end with the stack trace. Terraform v1.6.1 was built using github.com/hashicorp/hcl/v2@v2.18.1 and github.com/zclconf/go-cty@v1.14.1, so there should be no way to see the source of any other version in the stack trace.
micaroni
commented
11 months ago 2023-10-16T13:45:24.401-0600 [INFO] Terraform version: 1.6.1 2023-10-16T13:45:24.402-0600 [INFO] Go runtime version: go1.21.1 2023-10-16T13:45:24.402-0600 [INFO] CLI args: []string{"terraform", "plan"} 2023-10-16T13:45:24.403-0600 [INFO] CLI command args: []string{"plan"} 2023-10-16T13:45:24.428-0600 [INFO] backend/local: starting Plan operation 2023-10-16T13:45:24.546-0600 [INFO] backend/local: plan calling Plan
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Terraform crashed! This is always indicative of a bug within Terraform. Please report the crash with Terraform1 so that we can fix this.
When reporting bugs, please include your terraform version, the stack trace shown below, and any additional information which may help replicate the issue.
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
runtime error: invalid memory address or nil pointer dereference goroutine 46 [running]: runtime/debug.Stack() /Users/runner/hostedtoolcache/go/1.21.1/x64/src/runtime/debug/stack.go:24 +0x64 runtime/debug.PrintStack() /Users/runner/hostedtoolcache/go/1.21.1/x64/src/runtime/debug/stack.go:16 +0x1c github.com/hashicorp/terraform/internal/logging.PanicHandler() /Users/runner/work/terraform/terraform/internal/logging/panic.go:58 +0x164 panic({0x1072966e0?, 0x108eadda0?}) /Users/runner/hostedtoolcache/go/1.21.1/x64/src/runtime/panic.go:920 +0x26c github.com/hashicorp/terraform/internal/terraform.(NodePlannableResourceInstance).dataResourceExecute(0x14000a42bb0, {0x10779a490, 0x140007da1c0}) /Users/runner/work/terraform/terraform/internal/terraform/node_resource_plan_instance.go:94 +0x20c github.com/hashicorp/terraform/internal/terraform.(NodePlannableResourceInstance).Execute(0x151920e90?, {0x10779a490?, 0x140007da1c0?}, 0x80?) /Users/runner/work/terraform/terraform/internal/terraform/node_resource_plan_instance.go:76 +0x8c github.com/hashicorp/terraform/internal/terraform.(ContextGraphWalker).Execute(0x14000feefc0, {0x10779a490, 0x140007da1c0}, {0x151f0efe8, 0x14000a42bb0}) /Users/runner/work/terraform/terraform/internal/terraform/graph_walk_context.go:143 +0xa0 github.com/hashicorp/terraform/internal/terraform.(Graph).walk.func1({0x1076f8fe0, 0x14000a42bb0}) /Users/runner/work/terraform/terraform/internal/terraform/graph.go:78 +0x280 github.com/hashicorp/terraform/internal/dag.(Walker).walkVertex(0x140007c0600, {0x1076f8fe0, 0x14000a42bb0}, 0x14001209640) /Users/runner/work/terraform/terraform/internal/dag/walk.go:384 +0x2a8 created by github.com/hashicorp/terraform/internal/dag.(Walker).Update in goroutine 296 /Users/runner/work/terraform/terraform/internal/dag/walk.go:307 +0xb0c
jbardin
commented
11 months ago Thanks @micaroni, that matches the current release, and is a completely different stack trace from the first one. As for what is going on here, I'm not sure yet. In this case the trace logs may help give us more info. It appears that the config value is nil somehow, but this is executing a data source which can't exist without configuration so something unusual is going on.
Have there been any changes to the config since the last successful apply to help narrow things down, or was this a plan with an unchanged configuration which should resulted in nothing to apply?
micaroni
commented
11 months ago @jbardin I just want to be clear, I am not the original author but I have a similar problem.
Prior to this error, Terraform was simply throwing an error as usual. I had tried to create a large number of imported resources (~40) and it was behaving as expected when I ran `terraform plan'. But suddenly it crashed and continues to do so.
jbardin
commented
11 months ago @micaroni, sorry, I didn't catch that! Since this is a completely different stack trace from the original report, could you please file a new issue as it is not related. The known error it was giving would help too, as well as a reproduction if possible. Thanks!
internetti
commented
11 months ago Hi @internetti,
Can you verify that you have upgraded to Terraform v1.6.1? This stack trace you've shown here is from v1.6.0 and should be fixed in the current release.
Thanks!
hey @jbardin, thanks for the reply! I checked the logs from the setup step, and it does seem to be 1.6.1-1
2023-10-16T10:10:26.0383760Z Preparing to unpack .../terraform_1.6.1-1_amd64.deb ...
2023-10-16T10:10:26.1649431Z Unpacking terraform (1.6.1-1) ...Edit: I just tried setting the terraform version to 1.5.7-1, still get the same issue:
value is marked, so must be unmarked first
goroutine 722 [running]:
runtime/debug.Stack()
/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/debug/stack.go:24 +0x5e
runtime/debug.PrintStack()
/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/debug/stack.go:16 +0x13
github.com/hashicorp/terraform/internal/logging.PanicHandler()
/home/runner/work/terraform/terraform/internal/logging/panic.go:58 +0x13b
panic({0x2b92ec0?, 0x38c4c30?})
/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/panic.go:920 +0x270
github.com/zclconf/go-cty/cty.Value.assertUnmarked(...)
/home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.14.0/cty/marks.go:141
github.com/zclconf/go-cty/cty.Value.AsString({{{0x3902720?, 0xc000146bd1?}}, {0x2e1de40?, 0xc002cda528?}})
/home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.14.0/cty/value_ops.go:1385 +0x47
github.com/zclconf/go-cty/cty.Value.Range({{{0x3902720?, 0xc000146bd1?}}, {0x2e1de40?, 0xc002cda528?}})
/home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.14.0/cty/value_range.go:53 +0x285
github.com/hashicorp/hcl/v2/hclsyntax.(*ConditionalExpr).Value(0xc000569d50, 0xc002cda498)
/home/runner/go/pkg/mod/github.com/hashicorp/hcl/v2@v2.18.0/hclsyntax/expression.go:745 +0xd35
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalExpr(0xc002cb1170, {0x3902df8?, 0xc000569d50}, {{0x3902f48?, 0x5130520?}})
/home/runner/work/terraform/terraform/internal/lang/eval.go:175 +0x1bd
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateExpr(0x49?, {0x3902df8, 0xc000569d50}, {{0x3902f48?, 0x5130520?}}, {0x0?, 0x0?})
/home/runner/work/terraform/terraform/internal/terraform/eval_context_builtin.go:289 +0xab
github.com/hashicorp/terraform/internal/terraform.(*NodeLocal).Execute(0x7f406d1f3be0?, {0x391fb60, 0xc000bdd420}, 0x0?)
/home/runner/work/terraform/terraform/internal/terraform/node_local.go:158 +0x592
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0025fbd40, {0x391fb60, 0xc000bdd420}, {0x7f406d1ec3a8, 0xc0020fd590})
/home/runner/work/terraform/terraform/internal/terraform/graph_walk_context.go:143 +0xbe
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x2f593a0, 0xc0020fd590})
/home/runner/work/terraform/terraform/internal/terraform/graph.go:78 +0x375
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000a5fb00, {0x2f593a0, 0xc0020fd590}, 0xc002ca74c0)
/home/runner/work/terraform/terraform/internal/dag/walk.go:384 +0x2e5
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 521
/home/runner/work/terraform/terraform/internal/dag/walk.go:307 +0xde8
##[error]Script failed with exit code: 11@internetti,
It appears you are not executing the same Terraform binary you are unpacking in the first step. Neither v1.5.7 nor v1.6.1 can produce the stack trace you are showing here, since those versions and line numbers only align with v1.6.0.
internetti
commented
11 months ago Thank you, it was a versioning issue. I still don't know where the mismatch came from, but updating the version to 1.6.1 fixed it
github-actions[bot]
commented
9 months ago I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Terraform Version
Terraform Configuration Files
https://github.com/NorwegianRefugeeCouncil/core/tree/main/deploy/terraform
Debug Output
Expected Behavior
Should have deployed
Actual Behavior
Didn't deploy
Steps to Reproduce
terraform init terraform apply
Additional Context
No response
References
No response