search

brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 64 forks source link

Spreading heterogenous records back into an array causes "invalid union tag" #5339

Closed philrz closed 15 hours ago

philrz commented 2 days ago

tl;dr

$ echo '[{id:1},{c:2}]' | zq -z 'yield [...this]' -
panic: invalid union tag

goroutine 1 [running]:
github.com/brimdata/zed.(*TypeUnion).Untag(0xc0005b8ab0, {0xc000049399?, 0x85e78c5?, 0x30?})
    /Users/phil/work/zed/complex.go:341 +0x107
github.com/brimdata/zed/zson.(*elemHelper).add(0xc00054f708, {0xc000049399?, 0x0?, 0x0?})
    /Users/phil/work/zed/zson/formatter.go:476 +0x46
github.com/brimdata/zed/zson.(*Formatter).formatVector(0xc0004a8630, 0x0, {0x9bf71a8?, 0x1?}, {0x9bf22e0, 0x1}, {0x9f9a338, 0xc0005b8ab0}, {{0x9f9a2c0, 0xc0006c4c18}, ...}, ...)
    /Users/phil/work/zed/zson/formatter.go:440 +0x408
github.com/brimdata/zed/zson.(*Formatter).formatValue(0xc0004a8630, 0x0, {0x9f9a2c0, 0xc0006c4c18}, {0xc000049398, 0x11, 0x11}, 0x0, 0x0, 0x0)
    /Users/phil/work/zed/zson/formatter.go:173 +0xa5d
github.com/brimdata/zed/zson.(*Formatter).formatValueAndDecorate(0xc0004a8630, {0x9f9a2c0, 0xc0006c4c18?}, {0xc000049398, 0x11, 0x11})
    /Users/phil/work/zed/zson/formatter.go:146 +0x145
github.com/brimdata/zed/zson.(*Formatter).Format(0xc0004a8630, {{0x9f9a2c0?, 0xc0006c4c18?}, 0xc000049398?, 0x9d961c0?})
    /Users/phil/work/zed/zson/formatter.go:99 +0x92
github.com/brimdata/zed/zson.(*Formatter).FormatRecord(...)
    /Users/phil/work/zed/zson/formatter.go:76
github.com/brimdata/zed/zio/zsonio.(*Writer).Write(0xc0006c4888, {{0x9f9a2c0?, 0xc0006c4c18?}, 0xc000049398?, 0xc00054f901?})
    /Users/phil/work/zed/zio/zsonio/writer.go:34 +0x8e
github.com/brimdata/zed/zbuf.WriteBatch({0x51d2e508, 0xc0006c4888}, {0x9fa9038?, 0xc0006f8c80?})
    /Users/phil/work/zed/zbuf/batch.go:41 +0x84
github.com/brimdata/zed/zbuf.CopyMux(0xc00054fd58, {0x9f93720, 0xc0005b86c0})
    /Users/phil/work/zed/zbuf/mux.go:50 +0x106
github.com/brimdata/zed/cli/zq.(*Command).Run(0xc0004da1e0, {0xc00003e0e0, 0x2, 0x2})
    /Users/phil/work/zed/cli/zq/command.go:168 +0xb56
github.com/brimdata/zed/pkg/charm.path.run({0xc00062c328, 0x1, 0x1}, {0xc00003e0e0, 0x2, 0x0?})
    /Users/phil/work/zed/pkg/charm/path.go:11 +0x7f
github.com/brimdata/zed/pkg/charm.(*Spec).ExecRoot(0xae0fa40, {0xc00003e0d0, 0x3, 0x3})
    /Users/phil/work/zed/pkg/charm/charm.go:63 +0x3f
main.main()
    /Users/phil/work/zed/cmd/zq/main.go:11 +0x5b

Details

Repro is with Zed commit 4f551f2.

When both array values are of the same record type, there's no crash.

$ zq -version
Version: v1.18.0-29-g4f551f2b

$ echo '[{id:1},{id:2}]' | zq -z 'yield [...this]' -
[{id:1},{id:2}]
philrz commented 15 hours ago

Verified in Zed commit 91370cd.

The spread operation with the heterogenous records no longer causes a panic.

$ zq -version
Version: v1.18.0-31-g91370cde

$ echo '[{id:1},{c:2}]' | zq -z 'yield [...this]' -
[{id:1},{c:2}]

Thanks @mccanne!