search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.53k stars 9.52k forks source link

Crash cancelling destroy #27132

Closed innovate-invent closed 3 years ago

innovate-invent commented 3 years ago

Terraform Version

 terraform --version
Terraform v0.14.0-dev
+ provider registry.terraform.io/brinkmanlab/galaxy v0.2.10
+ provider registry.terraform.io/hashicorp/aws v3.18.0
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.3
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/time v0.6.0
+ provider registry.terraform.io/terraform-providers/docker v2.7.2

Your version of Terraform is out of date! The latest version
is 0.14.0. You can update by downloading from https://www.terraform.io/downloads.html

Terraform Configuration Files

...

Debug Output

End of console log ``` panic: close of nil channel 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: goroutine 5319 [running]: 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).stop(0xc000133a60) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:201 +0x71 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).Stop(0xc000133a60, 0x6e016e0, 0xc001028f60, 0xc001029020, 0xc000133a60, 0xc001028f60, 0xc0007c8ba0) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:215 +0xa1 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Stop_Handler(0x6049f60, 0xc000133a60, 0x6e016e0, 0xc001028f60, 0xc000ea75c0, 0x0, 0x6e016e0, 0xc001028f60, 0x0, 0x0) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:434 +0x217 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00, 0xc00178a510, 0xa5b8de8, 0x0, 0x0, 0x0) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x50a 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00, 0x0) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xcfd 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0004bb040, 0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00) 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xa1 2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1 2020-12-03T15:08:57.520-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x204 2020-12-03T15:08:57.543-0800 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.18.0/linux_amd64/terraform-provider-aws_v3.18.0_x5 pid=1496088 error="exit status 2" 2020/12/03 15:08:57 [TRACE] GRPCProvider: Stop 2020/12/03 15:08:57 [DEBUG] module.cloud.module.vpc.aws_subnet.public[1]: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Canceled desc = context canceled 2020-12-03T15:08:57.543-0800 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2020/12/03 15:08:57 [DEBUG] module.cloud.module.vpc.aws_subnet.public[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Canceled desc = context canceled 2020/12/03 15:08:57 [TRACE] EvalWriteState: writing current state object for module.cloud.module.vpc.aws_subnet.public[1] 2020/12/03 15:08:57 [TRACE] EvalWriteState: writing current state object for module.cloud.module.vpc.aws_subnet.public[0] 2020/12/03 15:08:57 [DEBUG] module.cloud.module.vpc.aws_subnet.public[2]: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing 2020/12/03 15:08:57 [TRACE] EvalWriteState: writing current state object for module.cloud.module.vpc.aws_subnet.public[2] 2020/12/03 15:08:57 [WARN] Early exit triggered by hook: *terraform.stopHook 2020/12/03 15:08:57 [TRACE] vertex "module.cloud.module.vpc.aws_subnet.public[1] (destroy)": visit complete 2020/12/03 15:08:57 [WARN] Early exit triggered by hook: *terraform.stopHook 2020/12/03 15:08:57 [WARN] Early exit triggered by hook: *terraform.stopHook 2020/12/03 15:08:57 [TRACE] vertex "module.cloud.module.vpc.aws_subnet.public[2] (destroy)": visit complete 2020/12/03 15:08:57 [TRACE] vertex "module.cloud.module.vpc.aws_subnet.public[0] (destroy)": visit complete 2020/12/03 15:08:57 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping 2020/12/03 15:08:57 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping 2020/12/03 15:08:57 [TRACE] dag/walk: upstream of "root" errored, so skipping 2020/12/03 15:08:57 [WARN] terraform: stop complete 2020/12/03 15:08:57 [TRACE] backend/local: graceful stop has completed 2020/12/03 15:08:57 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write 2020/12/03 15:08:57 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 457 2020/12/03 15:08:57 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate 2020/12/03 15:08:57 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info 2020/12/03 15:08:57 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock 2020-12-03T15:08:57.579-0800 [DEBUG] plugin: plugin exited 2020-12-03T15:08:57.579-0800 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2020-12-03T15:08:57.579-0800 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2020-12-03T15:08:57.580-0800 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2020-12-03T15:08:57.580-0800 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2020-12-03T15:08:57.587-0800 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/brinkmanlab/galaxy/0.2.10/linux_amd64/terraform-provider-galaxy_v0.2.10 pid=1496196 2020-12-03T15:08:57.587-0800 [DEBUG] plugin: plugin exited 2020-12-03T15:08:57.589-0800 [DEBUG] plugin: plugin process exited: path=/usr/bin/terraform pid=1495758 2020-12-03T15:08:57.589-0800 [DEBUG] plugin: plugin exited 2020-12-03T15:08:57.591-0800 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/1.13.3/linux_amd64/terraform-provider-kubernetes_v1.13.3_x4 pid=1496178 2020-12-03T15:08:57.592-0800 [DEBUG] plugin: plugin exited 2020-12-03T15:08:57.592-0800 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/1.13.3/linux_amd64/terraform-provider-kubernetes_v1.13.3_x4 pid=1496148 2020-12-03T15:08:57.592-0800 [DEBUG] plugin: plugin exited ```

Crash Output

crash.log

Expected Behavior

Exit normally

Actual Behavior

Crash

Steps to Reproduce

terraform destroy, wait 15min, CTRL-C

Additional Context

References

apparentlymart commented 3 years ago

Hi @innovate-invent! Thanks for reporting this.

The following is the stack trace at the end of the log, which I'm including here just because the log is very long and so it's easier to refer to the relevant part this way:

2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: panic: close of nil channel
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: 
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: goroutine 5319 [running]:
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).stop(0xc000133a60)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:201 +0x71
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).Stop(0xc000133a60, 0x6e016e0, 0xc001028f60, 0xc001029020, 0xc000133a60, 0xc001028f60, 0xc0007c8ba0)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:215 +0xa1
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Stop_Handler(0x6049f60, 0xc000133a60, 0x6e016e0, 0xc001028f60, 0xc000ea75c0, 0x0, 0x6e016e0, 0xc001028f60, 0x0, 0x0)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:434 +0x217
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00, 0xc00178a510, 0xa5b8de8, 0x0, 0x0, 0x0)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x50a
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00, 0x0)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xcfd
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0004bb040, 0xc00014e8c0, 0x6e25ea0, 0xc001b66000, 0xc000318d00)
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xa1
2020-12-03T15:08:57.519-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-12-03T15:08:57.520-0800 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x204

The structure of this log message suggests that it was the AWS provider process that crashed rather than the main Terraform process. However, the stack trace further suggests that the crash originated in the underlying provider protocol implementation library common to all providers, rather than in the AWS provider code itself.

Normally I'd respond to this by transferring the issue into the other repository so that the relevant team can see it, but that repository is not configured to give me access to perform that transfer so I'm going to have to poke the relevant team out of band to transfer it themselves. If you'd like though, another way to go would be to close this issue and open a similar issue in hashicorp/terraform-plugin-go, describing the same information. That might make the issue visible to that team sooner.

Thanks again!

jbardin commented 3 years ago

Thanks @innovate-invent,

This was already reported here https://github.com/hashicorp/terraform-plugin-go/issues/45, and fixed. The update should be coming soon in newer provider releases.

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.