search

brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.39k stars 64 forks source link

SEGV when "zq -o s3://" is downstream from failure #901

Closed philrz closed 4 years ago

philrz commented 4 years ago

I stumbled onto this quite by accident while verifying https://github.com/brimsec/zq/issues/771. I'm running zq commit 4e6680f.

The leading zq on the very front of the command line was unintentional, but triggered this stack dump. Obviously I should not have expected success since I made a mistake, but surely we'd not want the stack dump either.

$ zq echo '{"foo": "bar"}' | zq -o s3://zq-771/foo.zng -
{"foo": "bar"}: stat {"foo": "bar"}: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15bd64a]

goroutine 1 [running]:
io.(*PipeWriter).CloseWithError(...)
    /usr/local/opt/go/libexec/src/io/pipe.go:179
io.(*PipeWriter).Close(...)
    /usr/local/opt/go/libexec/src/io/pipe.go:169
github.com/brimsec/zq/pkg/s3io.(*Writer).Close(0xc000043f80, 0x0, 0x0)
    /Users/phil/work/zq/pkg/s3io/s3io.go:87 +0x2a
github.com/brimsec/zq/pkg/bufwriter.(*Writer).Close(0xc000322c20, 0x0, 0x0)
    /Users/phil/work/zq/pkg/bufwriter/writer.go:27 +0x68
github.com/brimsec/zq/zio.(*Writer).Close(0xc000322c40, 0x1, 0x8000000)
    /Users/phil/work/zq/zio/zio.go:54 +0x62
main.(*Command).Run(0xc0002e7380, 0xc0000320b0, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/work/zq/cmd/zq/zq.go:261 +0x8ba
github.com/mccanne/charm.(*instance).run(0xc00000fe20, 0xc000032090, 0x3, 0x3, 0x0, 0x0)
    /Users/phil/.go/pkg/mod/github.com/mccanne/charm@v0.0.3-0.20191224190439-b05e1b7b1be3/instance.go:53 +0x28c
github.com/mccanne/charm.(*Spec).ExecRoot(0x1ed4aa0, 0xc000032090, 0x3, 0x3, 0xc00031ff78, 0x10074cf, 0xc000100058, 0x0)
    /Users/phil/.go/pkg/mod/github.com/mccanne/charm@v0.0.3-0.20191224190439-b05e1b7b1be3/charm.go:77 +0x96
main.main()
    /Users/phil/work/zq/cmd/zq/main.go:9 +0x78

It appears like it may be unique to the new S3 output option, since there's no stack dump when I attempt to output to a local file.

$ zq echo '{"foo": "bar"}' | zq -o foo.zng -
{"foo": "bar"}: stat {"foo": "bar"}: no such file or directory
philrz commented 4 years ago

Verified in zq commit 8820a88. Now we get a helpful error message and no stack dump.

$ zq echo '{"foo": "bar"}' | zq -o s3://zq-771/foo.zng -
{"foo": "bar"}: stat {"foo": "bar"}: no such file or directory

Thanks @mattnibs!