search

brimdata / zync

Kafka connector to sync Zed lakes to and from Kafka topics
BSD 3-Clause "New" or "Revised" License
18 stars 3 forks source link

Crash during "zinger -h" when compiled with Go 1.17.1 on macOS #15

Closed philrz closed 2 years ago

philrz commented 2 years ago

After compiling/installing Zinger 16dc833 on my Mac, I tried to look at the current help text and saw the following crash.

$ zinger -h
NAME
    zinger - use zinger to receive, store, and transform zeek logs

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff20413c9e]

runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20413c9e
stack: frame={sp:0x7ffeefbff768, fp:0x7ffeefbff7b8} stack=[0x7ffeefb80808,0x7ffeefbff870)
0x00007ffeefbff668:  0x01007ffeefbff688  0x0000000000000004 
0x00007ffeefbff678:  0x000000000000001f  0x00007fff20413c9e 
0x00007ffeefbff688:  0x0b01dfacedebac1e  0x0000000000000001 
0x00007ffeefbff698:  0x00000000040349b1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff738 
0x00007ffeefbff6a8:  0x00000000049309cb  0x00007ffeefbff6f0 
0x00007ffeefbff6b8:  0x0000000004034c68 <runtime.fatalthrow.func1+0x0000000000000048>  0x0000000004eef620 
0x00007ffeefbff6c8:  0x0000000000000001  0x0000000000000001 
0x00007ffeefbff6d8:  0x00007ffeefbff738  0x00000000040349b1 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff6e8:  0x0000000004eef620  0x00007ffeefbff728 
0x00007ffeefbff6f8:  0x0000000004034bf0 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff708 
0x00007ffeefbff708:  0x0000000004034c20 <runtime.fatalthrow.func1+0x0000000000000000>  0x0000000004eef620 
0x00007ffeefbff718:  0x00000000040349b1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff738 
0x00007ffeefbff728:  0x00007ffeefbff758  0x00000000040349b1 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff738:  0x00007ffeefbff740  0x00000000040349e0 <runtime.throw.func1+0x0000000000000000> 
0x00007ffeefbff748:  0x000000000493e71f  0x000000000000002a 
0x00007ffeefbff758:  0x00007ffeefbff7a8  0x000000000404a156 <runtime.sigpanic+0x0000000000000396> 
0x00007ffeefbff768: <0x000000000493e71f  0x000000c0001003c0 
0x00007ffeefbff778:  0x00007ffeefbff7e8  0x0000000000000000 
0x00007ffeefbff788:  0x0000000004eef7c0  0x0000000000000000 
0x00007ffeefbff798:  0x000000c00019fbf8  0x000000000409e7e0 <syscall.libc_write_trampoline+0x0000000000000000> 
0x00007ffeefbff7a8:  0x00007ffeefbff7f0 !0x00007fff20413c9e 
0x00007ffeefbff7b8: >0x00007ffeefbff7f0  0x0000000004e34000 
0x00007ffeefbff7c8:  0x0000000000000703  0x0000000004441865 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005> 
0x00007ffeefbff7d8:  0x000000000406637f <runtime.syscall+0x000000000000001f>  0x000000c00019fd10 
0x00007ffeefbff7e8:  0x0000000004065cec <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c00019fce0 
0x00007ffeefbff7f8:  0x00000000040641f0 <runtime.asmcgocall+0x0000000000000070>  0x00007ffeefbff830 
0x00007ffeefbff808:  0x000000000400f1ce <runtime.persistentalloc.func1+0x000000000000002e>  0x0000000000000030 
0x00007ffeefbff818:  0x0000000000000010  0x0000000004f299e0 
0x00007ffeefbff828:  0x0000000000000348  0x000000c0000001a0 
0x00007ffeefbff838:  0x0000000004062309 <runtime.systemstack+0x0000000000000049>  0x0000000000000004 
0x00007ffeefbff848:  0x0000000004a629b0  0x0000000004eef620 
0x00007ffeefbff858:  0x00007ffeefbff8a8  0x0000000004062205 <runtime.mstart+0x0000000000000005> 
0x00007ffeefbff868:  0x00000000040621bd <runtime.rt0_go+0x000000000000013d> 
runtime.throw({0x493e71f, 0xc0001003c0})
    /usr/local/opt/go/libexec/src/runtime/panic.go:1198 +0x71
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20413c9e
stack: frame={sp:0x7ffeefbff768, fp:0x7ffeefbff7b8} stack=[0x7ffeefb80808,0x7ffeefbff870)
0x00007ffeefbff668:  0x01007ffeefbff688  0x0000000000000004 
0x00007ffeefbff678:  0x000000000000001f  0x00007fff20413c9e 
0x00007ffeefbff688:  0x0b01dfacedebac1e  0x0000000000000001 
0x00007ffeefbff698:  0x00000000040349b1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff738 
0x00007ffeefbff6a8:  0x00000000049309cb  0x00007ffeefbff6f0 
0x00007ffeefbff6b8:  0x0000000004034c68 <runtime.fatalthrow.func1+0x0000000000000048>  0x0000000004eef620 
0x00007ffeefbff6c8:  0x0000000000000001  0x0000000000000001 
0x00007ffeefbff6d8:  0x00007ffeefbff738  0x00000000040349b1 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff6e8:  0x0000000004eef620  0x00007ffeefbff728 
0x00007ffeefbff6f8:  0x0000000004034bf0 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff708 
0x00007ffeefbff708:  0x0000000004034c20 <runtime.fatalthrow.func1+0x0000000000000000>  0x0000000004eef620 
0x00007ffeefbff718:  0x00000000040349b1 <runtime.throw+0x0000000000000071>  0x00007ffeefbff738 
0x00007ffeefbff728:  0x00007ffeefbff758  0x00000000040349b1 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff738:  0x00007ffeefbff740  0x00000000040349e0 <runtime.throw.func1+0x0000000000000000> 
0x00007ffeefbff748:  0x000000000493e71f  0x000000000000002a 
0x00007ffeefbff758:  0x00007ffeefbff7a8  0x000000000404a156 <runtime.sigpanic+0x0000000000000396> 
0x00007ffeefbff768: <0x000000000493e71f  0x000000c0001003c0 
0x00007ffeefbff778:  0x00007ffeefbff7e8  0x0000000000000000 
0x00007ffeefbff788:  0x0000000004eef7c0  0x0000000000000000 
0x00007ffeefbff798:  0x000000c00019fbf8  0x000000000409e7e0 <syscall.libc_write_trampoline+0x0000000000000000> 
0x00007ffeefbff7a8:  0x00007ffeefbff7f0 !0x00007fff20413c9e 
0x00007ffeefbff7b8: >0x00007ffeefbff7f0  0x0000000004e34000 
0x00007ffeefbff7c8:  0x0000000000000703  0x0000000004441865 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005> 
0x00007ffeefbff7d8:  0x000000000406637f <runtime.syscall+0x000000000000001f>  0x000000c00019fd10 
0x00007ffeefbff7e8:  0x0000000004065cec <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c00019fce0 
0x00007ffeefbff7f8:  0x00000000040641f0 <runtime.asmcgocall+0x0000000000000070>  0x00007ffeefbff830 
0x00007ffeefbff808:  0x000000000400f1ce <runtime.persistentalloc.func1+0x000000000000002e>  0x0000000000000030 
0x00007ffeefbff818:  0x0000000000000010  0x0000000004f299e0 
0x00007ffeefbff828:  0x0000000000000348  0x000000c0000001a0 
0x00007ffeefbff838:  0x0000000004062309 <runtime.systemstack+0x0000000000000049>  0x0000000000000004 
0x00007ffeefbff848:  0x0000000004a629b0  0x0000000004eef620 
0x00007ffeefbff858:  0x00007ffeefbff8a8  0x0000000004062205 <runtime.mstart+0x0000000000000005> 
0x00007ffeefbff868:  0x00000000040621bd <runtime.rt0_go+0x000000000000013d> 
runtime.sigpanic()
    /usr/local/opt/go/libexec/src/runtime/signal_unix.go:719 +0x396

goroutine 1 [syscall]:
syscall.syscall(0x4441860, 0x0, 0x40087468, 0xc00019fda0)
    /usr/local/opt/go/libexec/src/runtime/sys_darwin.go:22 +0x3b fp=0xc00019fd10 sp=0xc00019fcf0 pc=0x406101b
syscall.syscall(0x0, 0xc00019fd88, 0xc00019fdc8, 0x404ec0f)
    <autogenerated>:1 +0x26 fp=0xc00019fd58 sp=0xc00019fd10 pc=0x4066b26
golang.org/x/sys/unix.ioctl(0x3, 0x40bb32e, 0x491f7e3)
    /Users/phil/.go/pkg/mod/golang.org/x/sys@v0.0.0-20210105210732-16f7687f5001/unix/zsyscall_darwin_amd64.go:689 +0x39 fp=0xc00019fd88 sp=0xc00019fd58 pc=0x44414f9
golang.org/x/sys/unix.IoctlGetWinsize(...)
    /Users/phil/.go/pkg/mod/golang.org/x/sys@v0.0.0-20210105210732-16f7687f5001/unix/ioctl.go:66
golang.org/x/term.getSize(0x491f2a3)
    /Users/phil/.go/pkg/mod/golang.org/x/term@v0.0.0-20210317153231-de623e64d2a6/term_unix.go:61 +0x2c fp=0xc00019fdb8 sp=0xc00019fd88 pc=0x444196c
golang.org/x/term.GetSize(...)
    /Users/phil/.go/pkg/mod/golang.org/x/term@v0.0.0-20210317153231-de623e64d2a6/term.go:50
github.com/brimdata/zed/pkg/terminal.Width()
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v0.30.0/pkg/terminal/terminal.go:17 +0x55 fp=0xc00019fdd8 sp=0xc00019fdb8 pc=0x4441a95
github.com/brimdata/zed/pkg/charm.helpDesc({0x491fec6, 0x492082b}, {0x4946fb3, 0x38})
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v0.30.0/pkg/charm/help.go:64 +0xb4 fp=0xc00019fe70 sp=0xc00019fdd8 pc=0x4510054
github.com/brimdata/zed/pkg/charm.displayHelp({0xc0000100e8, 0x1, 0x1}, 0x1)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v0.30.0/pkg/charm/help.go:134 +0xb2 fp=0xc00019fec8 sp=0xc00019fe70 pc=0x4510972
github.com/brimdata/zed/pkg/charm.(*Spec).ExecRoot(0xc0000001a0, {0xc000020050, 0x1, 0x1})
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v0.30.0/pkg/charm/charm.go:70 +0xd6 fp=0xc00019ff28 sp=0xc00019fec8 pc=0x450f8f6
main.main()
    /Users/phil/work/zinger/cmd/zinger/main.go:21 +0x5f fp=0xc00019ff80 sp=0xc00019ff28 pc=0x455727f
runtime.main()
    /usr/local/opt/go/libexec/src/runtime/proc.go:255 +0x227 fp=0xc00019ffe0 sp=0xc00019ff80 pc=0x4037047
runtime.goexit()
    /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00019ffe8 sp=0xc00019ffe0 pc=0x40644e1

I actually had seen a similar crash when attempting a zinger consume command line in the dust-off branch before it merged. I assume they share a root cause, but if it can be triggered with just zinger -h that seems the easy one to start with.

@nwt and I had some back & forth trying to narrow down the root cause of this, and it looks like it's related to Go version:

I get the crash with Go 1.17.1 but not with 1.16.8.

It looks like the problem may be unique to macOS, though. I just compiled with Go 1.17.1 on a Linux Ubuntu 18.04 VM and zinger -h ran successfully without crashing.

nwt commented 2 years ago

Looks like a Go 1.17 bug triggered by system calls through golang.org/x/sys/unix when using cgo. (It affects zinger because github.com/confluentinc/confluent-kafka-go uses cgo.)

$ uname -v
Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64

$ go version
go version go1.17.1 darwin/amd64

$ cat main.go
package main

import "C"
import "golang.org/x/sys/unix"

func main() {
    unix.CloseOnExec(0)
}

$ go run main.go
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff20440c9e]

runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20440c9e
stack: frame={sp:0x7ffeefbff5c8, fp:0x7ffeefbff618} stack=[0x7ffeefb80668,0x7ffeefbff6d0)
0x00007ffeefbff4c8:  0x01007ffeefbff4e8  0x0000000000000004
0x00007ffeefbff4d8:  0x000000000000001f  0x00007fff20440c9e
0x00007ffeefbff4e8:  0x0b01dfacedebac1e  0x0000000000000001
0x00007ffeefbff4f8:  0x000000000402c571 <runtime.throw+0x0000000000000071>  0x00007ffeefbff598
0x00007ffeefbff508:  0x000000000406db60  0x00007ffeefbff550
0x00007ffeefbff518:  0x000000000402c828 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000040dbae0
0x00007ffeefbff528:  0x0000000000000001  0x0000000000000001
0x00007ffeefbff538:  0x00007ffeefbff598  0x000000000402c571 <runtime.throw+0x0000000000000071>
0x00007ffeefbff548:  0x00000000040dbae0  0x00007ffeefbff588
0x00007ffeefbff558:  0x000000000402c7b0 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff568
0x00007ffeefbff568:  0x000000000402c7e0 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000040dbae0
0x00007ffeefbff578:  0x000000000402c571 <runtime.throw+0x0000000000000071>  0x00007ffeefbff598
0x00007ffeefbff588:  0x00007ffeefbff5b8  0x000000000402c571 <runtime.throw+0x0000000000000071>
0x00007ffeefbff598:  0x00007ffeefbff5a0  0x000000000402c5a0 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff5a8:  0x000000000406efc5  0x000000000000002a
0x00007ffeefbff5b8:  0x00007ffeefbff608  0x000000000403ff56 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff5c8: <0x000000000406efc5  0x00007ffeefbff630
0x00007ffeefbff5d8:  0x00007ffeefbff648  0x0000000004020886 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff5e8:  0x000000c0000ac000  0x0000000000002000
0x00007ffeefbff5f8:  0x000000c000000008  0x0000000000000000
0x00007ffeefbff608:  0x00007ffeefbff650 !0x00007fff20440c9e
0x00007ffeefbff618: >0x00007ffeefbff650  0x00000000040c8000
0x00007ffeefbff628:  0x0000000000000070  0x000000000405c145 <golang.org/x/sys/unix.libc_fcntl_trampoline+0x0000000000000005>
0x00007ffeefbff638:  0x000000000405585f <runtime.syscall+0x000000000000001f>  0x000000c000058ed8
0x00007ffeefbff648:  0x00007ffeefbff690  0x000000c000058ea8
0x00007ffeefbff658:  0x0000000004053750 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000001
0x00007ffeefbff668:  0x000000c000080d00  0x0d007ffeefbff690
0x00007ffeefbff678:  0x0000000004036dc5 <runtime.malg.func1+0x0000000000000025>  0x000000c000008000
0x00007ffeefbff688:  0x0000000000000180  0x000000c0000001a0
0x00007ffeefbff698:  0x0000000004053529 <runtime.systemstack+0x0000000000000049>  0x0000000000000004
0x00007ffeefbff6a8:  0x0000000004080cf8  0x00000000040dbae0
0x00007ffeefbff6b8:  0x00007ffeefbff700  0x0000000004053425 <runtime.mstart+0x0000000000000005>
0x00007ffeefbff6c8:  0x00000000040533dd <runtime.rt0_go+0x000000000000013d>
runtime.throw({0x406efc5, 0x7ffeefbff630})
    /usr/local/Cellar/go/1.17.1/libexec/src/runtime/panic.go:1198 +0x71
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20440c9e
stack: frame={sp:0x7ffeefbff5c8, fp:0x7ffeefbff618} stack=[0x7ffeefb80668,0x7ffeefbff6d0)
0x00007ffeefbff4c8:  0x01007ffeefbff4e8  0x0000000000000004
0x00007ffeefbff4d8:  0x000000000000001f  0x00007fff20440c9e
0x00007ffeefbff4e8:  0x0b01dfacedebac1e  0x0000000000000001
0x00007ffeefbff4f8:  0x000000000402c571 <runtime.throw+0x0000000000000071>  0x00007ffeefbff598
0x00007ffeefbff508:  0x000000000406db60  0x00007ffeefbff550
0x00007ffeefbff518:  0x000000000402c828 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000040dbae0
0x00007ffeefbff528:  0x0000000000000001  0x0000000000000001
0x00007ffeefbff538:  0x00007ffeefbff598  0x000000000402c571 <runtime.throw+0x0000000000000071>
0x00007ffeefbff548:  0x00000000040dbae0  0x00007ffeefbff588
0x00007ffeefbff558:  0x000000000402c7b0 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff568
0x00007ffeefbff568:  0x000000000402c7e0 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000040dbae0
0x00007ffeefbff578:  0x000000000402c571 <runtime.throw+0x0000000000000071>  0x00007ffeefbff598
0x00007ffeefbff588:  0x00007ffeefbff5b8  0x000000000402c571 <runtime.throw+0x0000000000000071>
0x00007ffeefbff598:  0x00007ffeefbff5a0  0x000000000402c5a0 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff5a8:  0x000000000406efc5  0x000000000000002a
0x00007ffeefbff5b8:  0x00007ffeefbff608  0x000000000403ff56 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff5c8: <0x000000000406efc5  0x00007ffeefbff630
0x00007ffeefbff5d8:  0x00007ffeefbff648  0x0000000004020886 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff5e8:  0x000000c0000ac000  0x0000000000002000
0x00007ffeefbff5f8:  0x000000c000000008  0x0000000000000000
0x00007ffeefbff608:  0x00007ffeefbff650 !0x00007fff20440c9e
0x00007ffeefbff618: >0x00007ffeefbff650  0x00000000040c8000
0x00007ffeefbff628:  0x0000000000000070  0x000000000405c145 <golang.org/x/sys/unix.libc_fcntl_trampoline+0x0000000000000005>
0x00007ffeefbff638:  0x000000000405585f <runtime.syscall+0x000000000000001f>  0x000000c000058ed8
0x00007ffeefbff648:  0x00007ffeefbff690  0x000000c000058ea8
0x00007ffeefbff658:  0x0000000004053750 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000001
0x00007ffeefbff668:  0x000000c000080d00  0x0d007ffeefbff690
0x00007ffeefbff678:  0x0000000004036dc5 <runtime.malg.func1+0x0000000000000025>  0x000000c000008000
0x00007ffeefbff688:  0x0000000000000180  0x000000c0000001a0
0x00007ffeefbff698:  0x0000000004053529 <runtime.systemstack+0x0000000000000049>  0x0000000000000004
0x00007ffeefbff6a8:  0x0000000004080cf8  0x00000000040dbae0
0x00007ffeefbff6b8:  0x00007ffeefbff700  0x0000000004053425 <runtime.mstart+0x0000000000000005>
0x00007ffeefbff6c8:  0x00000000040533dd <runtime.rt0_go+0x000000000000013d>
runtime.sigpanic()
    /usr/local/Cellar/go/1.17.1/libexec/src/runtime/signal_unix.go:719 +0x396

goroutine 1 [syscall]:
syscall.syscall(0x405c140, 0x0, 0x2, 0x1)
    /usr/local/Cellar/go/1.17.1/libexec/src/runtime/sys_darwin.go:22 +0x3b fp=0xc000058ed8 sp=0xc000058eb8 pc=0x405245b
syscall.syscall(0x200000003, 0xc0000001a0, 0xc000058f70, 0x4004e99)
    <autogenerated>:1 +0x26 fp=0xc000058f20 sp=0xc000058ed8 pc=0x4055e26
golang.org/x/sys/unix.fcntl(0x0, 0x4060e20, 0xc0000001a0)
    /Users/noah/looky/pkg/mod/golang.org/x/sys@v0.0.0-20210105210732-16f7687f5001/unix/zsyscall_darwin_amd64.go:660 +0x39 fp=0xc000058f58 sp=0xc000058f20 pc=0x405bdb9
golang.org/x/sys/unix.CloseOnExec(...)
    /Users/noah/looky/pkg/mod/golang.org/x/sys@v0.0.0-20210105210732-16f7687f5001/unix/syscall_unix.go:388
main.main()
    /Users/noah/looky/zinger/main.go:7 +0x25 fp=0xc000058f80 sp=0xc000058f58 pc=0x405c1c5
runtime.main()
    /usr/local/Cellar/go/1.17.1/libexec/src/runtime/proc.go:255 +0x227 fp=0xc000058fe0 sp=0xc000058f80 pc=0x402ec07
runtime.goexit()
    /usr/local/Cellar/go/1.17.1/libexec/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x4053a21
exit status 2
nwt commented 2 years ago

This is fixed in Go 1.17.3. I can't reproduce this with Go 1.17.1 on macOS Monterey 12.0.1 (kernel 21.1.0).