Packer intermittent crashes: panic: ConfigSpec failed: connection is shut down [recovered]

buchs commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

Intermittent crashes with packer build. See subject/title. Packer requested I report this crash:

!!!!!!!!!!!!!!!!!!!!!!!!!!! PACKER CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Packer crashed! This is always indicative of a bug within Packer.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Packer[1] so that we can fix this.

[1]: https://github.com/hashicorp/packer/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! PACKER CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Reproduction Steps

packer build .

Packer version

1.8.5

Simplified Packer Template

Need private way to provide this. Also the crash.log.

Operating system and Environment details

Ubuntu 22.04, x86_64

Log Fragments and crash.log files

see above.

Set the env var PACKER_LOG=1 for maximum log detail.

tail of crash log:

2023/02/27 17:57:27 ConfigSpec failed: connection is shut down
panic: ConfigSpec failed: connection is shut down [recovered]
        panic: ConfigSpec failed: connection is shut down

goroutine 772 [running]:
log.Panic({0xc0014ccf88?, 0x1?, 0xc000bfc460?})
        /opt/hostedtoolcache/go/1.18.9/x64/src/log/log.go:385 +0x65
github.com/hashicorp/packer/packer.(*cmdProvisioner).checkExit(0xc0004ab730?, {0x4017300, 0xc00088ed30}, 0x0)
        /home/runner/work/packer/packer/packer/cmd_provisioner.go:47 +0x7f
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec.func1()
        /home/runner/work/packer/packer/packer/cmd_provisioner.go:19 +0x39
panic({0x4017300, 0xc00088ed30})
        /opt/hostedtoolcache/go/1.18.9/x64/src/runtime/panic.go:838 +0x207
github.com/hashicorp/packer-plugin-sdk/rpc.(*commonClient).ConfigSpec(0xc001095620)
        /home/runner/go/pkg/mod/github.com/hashicorp/packer-plugin-sdk@v0.3.2/rpc/common.go:44 +0x297
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec(0x48fe4c0?)
        /home/runner/work/packer/packer/packer/cmd_provisioner.go:22 +0x65
github.com/hashicorp/packer/hcl2template.decodeHCL2Spec({0x5927740, 0xc000c60600}, 0x50e5c69?, {0x7f89868582d8?, 0xc00109bce0?})
        /home/runner/work/packer/packer/hcl2template/decode.go:17 +0x39
github.com/hashicorp/packer/hcl2template.(*HCL2Provisioner).HCL2Prepare(0xc0010cc450, 0xc0010a0ea0?)
        /home/runner/work/packer/packer/hcl2template/types.hcl_provisioner.go:51 +0x447
github.com/hashicorp/packer/hcl2template.(*HCL2Provisioner).Provision(0xc0010cc450, {0x5926da0, 0xc000cc0600}, {0x592bb88, 0xc0010a0e70}, {0x5929f10, 0xc000cceee0}, 0xc000cceee0?)
        /home/runner/work/packer/packer/hcl2template/types.hcl_provisioner.go:71 +0x4e
github.com/hashicorp/packer/packer.(*ProvisionHook).Run(0xc000665160?, {0x5926da0, 0xc000cc0600}, {0x7f89ad6fd108?, 0x20?}, {0x592bb88, 0xc0010a0e70}, {0x5929f10?, 0xc000cceee0}, {0x4335480, ...})
        /home/runner/work/packer/packer/packer/provisioner.go:122 +0x1cb
github.com/hashicorp/packer-plugin-sdk/packer.(*DispatchHook).Run(0x0?, {0x5926da0, 0xc000cc0600}, {0xc000665160, 0x10}, {0x592bb88, 0xc0010a0e70}, {0x5929f10, 0xc000cceee0}, {0x4335480, ...})
        /home/runner/go/pkg/mod/github.com/hashicorp/packer-plugin-sdk@v0.3.2/packer/hook.go:47 +0x172
github.com/hashicorp/packer-plugin-sdk/rpc.(*HookServer).Run(0xc000c3ddc0, 0xc0010a0c60, 0xc00088ea20)
        /home/runner/go/pkg/mod/github.com/hashicorp/packer-plugin-sdk@v0.3.2/rpc/hook.go:75 +0x263
reflect.Value.call({0xc000bfe2a0?, 0xc001553d28?, 0x13?}, {0x50e42f9, 0x4}, {0xc0007c6ef8, 0x3, 0x3?})
        /opt/hostedtoolcache/go/1.18.9/x64/src/reflect/value.go:556 +0x845
reflect.Value.Call({0xc000bfe2a0?, 0xc001553d28?, 0x0?}, {0xc0010f56f8, 0x3, 0x3})
        /opt/hostedtoolcache/go/1.18.9/x64/src/reflect/value.go:339 +0xbf
net/rpc.(*service).call(0xc000c3de00, 0x0?, 0xc000a58fc0?, 0xc000664f80, 0xc001530480, 0xc0010f57d0?, {0x3e6bbe0?, 0xc0010a0c60?, 0xc0006ba540?}, {0x3edfbe0, ...}, ...)
        /opt/hostedtoolcache/go/1.18.9/x64/src/net/rpc/server.go:381 +0x226
created by net/rpc.(*Server).ServeCodec
        /opt/hostedtoolcache/go/1.18.9/x64/src/net/rpc/server.go:478 +0x3fe

nywilken commented 1 year ago

Hi @buchs thanks for reporting this issue, and I'm sorry to hear you are running into this crash. Could provide a template that we can use to reproduce the issue please.

Is there anything specific about the environment that we should know about?

I do wonder if communication to the provisioner is being terminated prematurely. Any details about the environment and the configuration would be helpful to better understand how to reproduce. Thanks for the report and time.

Wilken

buchs commented 1 year ago

Hi Wilken,

I'll be happy to send along the template, if there is a non-public way for me to share it.

I was building from a AWS T3.micro instance running Ubuntu 22.04. My initial attempts were building 8 images in parallel - 2 AWS regions, 2 processor arch and 2 versions of Ubuntu 20.04 and 22.04.

When I split up the build into 2 sets of 4 images and added a 10-15 second delay between them it is now not crashing in a consistent manner. Each set of 4 images takes about 30 minutes to build.

Kevin

hashicorp / packer