search

brimdata / zync

Kafka connector to sync Zed lakes to and from Kafka topics
BSD 3-Clause "New" or "Revised" License
17 stars 3 forks source link

"slice bounds out of range" error when running through demo #91

Closed philrz closed 1 year ago

philrz commented 2 years ago

Repro is with zync commit d87d49b and zed commit 28029f0.

Running through the Demo steps, first the prep in one shell:

$ zync version
d87d49b

$ zed -version
Version: v1.1.0-44-g28029f06

$ zed serve -lake scratch
{"level":"info","ts":1657068901.776302,"msg":"Open files limit raised","limit":61440}
...

And now the remaining demo steps:

$ zed create -orderby kafka.offset Raw
pool created: Raw 2BY2tQIYQcezbrknSvS9BzoGPR1

$ zed create -orderby kafka.offset Staging
pool created: Staging 2BY2tkpvAjj1D35RPDutwCwi57I

$ zed load -use Raw@main demo/batch-1.zson
(11/1) 1023B/1023B 1023B/s 100.00%
2BY2uS0mXPcvYiy06d2FS6hQTKm committed

$ zync etl demo/invoices.yaml
commit 2BY2vGKY89NajJDhxqp2WSAiUBB 0 records
ETL'd 4 records

$ zed load -use Raw@main demo/batch-2.zson
(11/1) 567B/567B 567B/s 100.00%
2BY2wOsYMEyuA6Sfqbxt3zi80fm committed

$ zync etl demo/invoices.yaml
panic: runtime error: slice bounds out of range [123:1]

goroutine 1 [running]:
github.com/brimdata/zed/zcode.(*Iter).Next(0xc0000b9540?)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zcode/iter.go:34 +0x1af
github.com/brimdata/zed/zson.(*Formatter).formatRecord(0xc0000b9540, 0xc00059b568?, 0x1089265?, {0xc000a00003?, 0x100000114dba5?, 0x3b0498c8?}, 0x60?, 0x6d?)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:433 +0x7d7
github.com/brimdata/zed/zson.(*Formatter).formatValue(0xc0000b9540, 0xc0002bc130?, {0x1b3f7c8?, 0xc00063c4b0?}, {0xc000a00003, 0x1, 0x7fffd}, 0x0, 0x1, 0x1)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:175 +0x1ec
github.com/brimdata/zed/zson.(*Formatter).formatRecord(0xc0000b9540, 0x0?, 0xc00063c8a0?, {0xc000a00002?, 0x2f?, 0x7fffe?}, 0xa0?, 0x0?)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:433 +0x812
github.com/brimdata/zed/zson.(*Formatter).formatValue(0xc0000b9540, 0xc00063c8a0?, {0x1b3f7c8?, 0xc00063c8a0?}, {0xc000a00002, 0x2f, 0x7fffe}, 0x0, 0x1, 0x0)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:175 +0x1ec
github.com/brimdata/zed/zson.(*Formatter).formatValueAndDecorate(0xc0000b9540, {0x1b3f7c8, 0xc00063c8a0}, {0xc000a00002, 0x2f, 0x7fffe})
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:146 +0x105
github.com/brimdata/zed/zson.(*Formatter).Format(0xc0000b9540, 0x1154be8?)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:113 +0x7b
github.com/brimdata/zed/zson.FormatValue(0xc0001380a0?)
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/zson/formatter.go:86 +0xae
github.com/brimdata/zync/etl.insertOffsets({0x1b426b8, 0xc000093a00}, 0xc000128060, {0x189dac0?, 0xc00063d2f0?}, {0x1b43520?, 0xc00000e180?}, 0xc000331b78?)
    /Users/phil/work/zync/etl/pipeline.go:146 +0x125
github.com/brimdata/zync/etl.(*Pipeline).writeToOutputPool(0xc00013e1c0, {0x1b426b8, 0xc000093a00}, 0xc00000e180)
    /Users/phil/work/zync/etl/pipeline.go:97 +0x94
github.com/brimdata/zync/etl.(*Pipeline).Run(0xc00013e1c0, {0x1b426b8, 0xc000093a00})
    /Users/phil/work/zync/etl/pipeline.go:75 +0x105
github.com/brimdata/zync/cmd/zync/etl.(*Command).Run(0xc0000d2380, {0xc0000321d0?, 0x1?, 0x0?})
    /Users/phil/work/zync/cmd/zync/etl/command.go:82 +0x1e9
github.com/brimdata/zed/pkg/charm.path.run({0xc0002c6bc0?, 0x2, 0x2}, {0xc0000321d0?, 0x1, 0x0?})
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/pkg/charm/path.go:11 +0x8b
github.com/brimdata/zed/pkg/charm.(*Spec).ExecRoot(0xc0000021a0?, {0xc0000321c0, 0x2, 0x2})
    /Users/phil/.go/pkg/mod/github.com/brimdata/zed@v1.1.1-0.20220617194348-ccabb55b2bc9/pkg/charm/charm.go:63 +0x4a
main.main()
    /Users/phil/work/zync/cmd/zync/main.go:19 +0x5e
philrz commented 1 year ago

Verified with zync commit 8387674 and Zed commit 3b24a114.

Repeating the demo steps, we now see that the final ETL command no longer crashes.

First, in one shell:

$ zync version
8387674

$ zed -version
Version: v1.2.0-1-g3b24a114

$ zed serve -lake scratch
{"level":"info","ts":1657844896.643243,"msg":"Open files limit raised","limit":61440}
...

And the remaining demo steps:

$ zed create -orderby kafka.offset Raw
pool created: Raw 2BxPkcPQow2KmXA2yf71hTGr1pW

$ zed create -orderby kafka.offset Staging
pool created: Staging 2BxPl8ikmUAVEpQgLlyNlbwXA64

$ zed load -use Raw@main demo/batch-1.zson
(11/1) 1023B/1023B 1023B/s 100.00%
2BxPmuNOlAoKkKL505rqxvz7KgW committed

$ zync etl demo/invoices.yaml
commit 2BxPnh0R99r4NxJn5VqG491fEIb 0 records
ETL'd 4 records

$ zed load -use Raw@main demo/batch-2.zson
(11/1) 567B/567B 567B/s 100.00%
2BxPomtdSXvaANkMQa93Cpag06L committed

$ zync etl demo/invoices.yaml
commit 2BxPpNO693Be6ufcttNhMoebqHA 0 records
ETL'd 2 records

Thanks @nwt!