search

brimdata / zed

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 67 forks source link

Zed lake service crash during concurrent pool renames #5261

Closed philrz closed 3 weeks ago

philrz commented 1 month ago

tl;dr

A Zed crash can be trigged if multiple pool rename operations are initiated close together.

Details

Repro is with Zed commit f8fb140.

I became aware of this issue because of a a recent Zui test failure observed in CI. I have a Zui branch pools-api-zed-crash that isolates the failure and can reproduce the crash locally on my laptop by looping:

while true; do yarn test-pools-api ; done

However, after studying what the test is doing, I also have this repro script that'll trigger the crash using just Zed CLI tools. First start a lake service:

$ zed -version
Version: v1.17.0-47-gf8fb1406

$ zed -lake lake serve
{"level":"info","ts":1725552565.296819,"logger":"core","msg":"Started","auth_enabled":false,"root":"file:///Users/phil/lake","version":"v1.17.0-47-gf8fb1406"}

The repro script:

#!/bin/bash
set -euox pipefail
NUM=8

for n in $(seq $NUM)
do
  zed create foo$n &
done
wait

sleep 1

for n in $(seq $NUM)
do
  zed rename foo$n bar$n &
done
wait

for n in $(seq $NUM)
do
  zed drop -f bar$n
done

When running it, the crash dump as the Zed lake service exits:

fatal error: concurrent map writes

goroutine 818 [running]:
github.com/brimdata/zed/zson.(*binder).enter(...)
    /Users/phil/work/zed/zson/marshal.go:1082
github.com/brimdata/zed/zson.(*binder).enterBinding(0xc000594958, {{0x2850331?, 0x269c260?}, {0x27930c0?, 0x3a766e0?}})
    /Users/phil/work/zed/zson/marshal.go:1116 +0x165
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).NamedBindings(...)
    /Users/phil/work/zed/zson/marshal.go:696
github.com/brimdata/zed/lake/pools.(*Config).UnmarshalZNG(0xc000527380, 0xc000594948, {0x0?, 0x0?})
    /Users/phil/work/zed/lake/pools/config.go:96 +0xaf
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000527380?, 0xc000fb9a40?}, {0x27a63c0?, 0xc000527380?, 0x7fff6?})
    /Users/phil/work/zed/zson/marshal.go:712 +0x2643
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000f0c100?, 0xc000fb9a40?}, {0x263b120?, 0xc000f0c100?, 0x263b120?})
    /Users/phil/work/zed/zson/marshal.go:791 +0x1f39
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeRecord(0x2b51500?, {0x100000c000f022d0, 0x4000002900000001}, {0x269cbc0?, 0xc000f0c100?, 0x0?})
    /Users/phil/work/zed/zson/marshal.go:978 +0x7d7
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000f0c100?, 0xc0006a8420?}, {0x269cbc0?, 0xc000f0c100?, 0x7fff7?})
    /Users/phil/work/zed/zson/marshal.go:759 +0x252e
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000f0c0f0?, 0x0?}, {0x2532780?, 0xc000f0c0f0?, 0x10?})
    /Users/phil/work/zed/zson/marshal.go:791 +0x1f39
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).Unmarshal(0xc001690120?, {0xc000fb8000?, 0xc000f94db0?}, {0x2532780?, 0xc000f0c0f0?})
    /Users/phil/work/zed/zson/marshal.go:674 +0x6b
github.com/brimdata/zed/lake/journal.(*Store).load(0xc0004b9320, {0x2b60a10, 0xc0005f88d0})
    /Users/phil/work/zed/lake/journal/store.go:129 +0x50b
github.com/brimdata/zed/lake/journal.(*Store).commit(0xc0004b9320, {0x2b60a10, 0xc0005f88d0}, 0xc00096d238, {0xc000c27218, 0x2, 0xc0005a02c0?})
    /Users/phil/work/zed/lake/journal/store.go:343 +0x13b
github.com/brimdata/zed/lake/journal.(*Store).Move(0xc0004b9320, {0x2b60a10, 0xc0005f88d0}, {0xc000015d70, 0x4}, {0x2b49880?, 0xc0027ace70})
    /Users/phil/work/zed/lake/journal/store.go:297 +0x190
github.com/brimdata/zed/lake/pools.(*Store).Rename(0xc0005b0060, {0x2b60a10, 0xc0005f88d0}, {0x13, 0x67, 0x8a, 0xb4, 0x92, 0x3d, 0x86, ...}, ...)
    /Users/phil/work/zed/lake/pools/store.go:91 +0xde
github.com/brimdata/zed/lake.(*Root).RenamePool(...)
    /Users/phil/work/zed/lake/root.go:314
github.com/brimdata/zed/service.handlePoolPut(0xc0005a9560, 0xc000f8ae10, 0xc000a1be20)
    /Users/phil/work/zed/service/handlers.go:277 +0x145
github.com/brimdata/zed/service.(*Core).authhandle.(*Core).handler.func2({0x2b5a900?, 0xc00158ae20?}, 0xc000822fc0?)
    /Users/phil/work/zed/service/core.go:198 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005c49a0?, {0x2b5a900?, 0xc00158ae20?}, 0xc001611600?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x2b5a900, 0xc00158ae20}, 0xc001611600)
    /Users/phil/.go/pkg/mod/github.com/rs/cors@v1.8.0/cors.go:219 +0x17e
net/http.HandlerFunc.ServeHTTP(0xc000c274a8?, {0x2b5a900?, 0xc00158ae20?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.panicCatchMiddleware.func6.1({0x2b5a900?, 0xc00158ae20?}, 0x10d120f?)
    /Users/phil/work/zed/service/middleware.go:92 +0x7a
net/http.HandlerFunc.ServeHTTP(0x2b5aa80?, {0x2b5a900?, 0xc00158ae20?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.accessLogMiddleware.func5.1({0x2b5aa80, 0xc00195c0e0}, 0xc001611600)
    /Users/phil/work/zed/service/middleware.go:72 +0x8b4
net/http.HandlerFunc.ServeHTTP(0xc001611500?, {0x2b5aa80?, 0xc00195c0e0?}, 0x2b42470?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.requestIDMiddleware.func4.1({0x2b5aa80, 0xc00195c0e0}, 0xc001611500)
    /Users/phil/work/zed/service/middleware.go:28 +0x14b
net/http.HandlerFunc.ServeHTTP(0xc001611300?, {0x2b5aa80?, 0xc00195c0e0?}, 0xc000c27aa8?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005dc240, {0x2b5aa80, 0xc00195c0e0}, 0xc001611200)
    /Users/phil/.go/pkg/mod/github.com/gorilla/mux@v1.7.5-0.20200711200521-98cb6bf42e08/mux.go:210 +0x1c5
github.com/brimdata/zed/service.(*Core).ServeHTTP(0xc0005a9560, {0x2b5aa80, 0xc00195c0e0}, 0xc00195c001?)
    /Users/phil/work/zed/service/core.go:220 +0xa5
net/http.serverHandler.ServeHTTP({0x2b51be0?}, {0x2b5aa80?, 0xc00195c0e0?}, 0x6?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2943 +0x8e
net/http.(*conn).serve(0xc0004f2900, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2014 +0x5f4
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000595068?)
    /usr/local/opt/go@1.21/libexec/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x268e6a0?)
    /usr/local/opt/go@1.21/libexec/src/sync/waitgroup.go:116 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc0005a0e40)
    /Users/phil/.go/pkg/mod/golang.org/x/sync@v0.4.0/errgroup/errgroup.go:53 +0x25
github.com/brimdata/zed/cmd/zed/serve.(*Command).Run(0xc0000f7560, {0x2b48de0?, 0x0?, 0x0?})
    /Users/phil/work/zed/cmd/zed/serve/command.go:138 +0x908
github.com/brimdata/zed/pkg/charm.path.run({0xc0001bf110?, 0x2, 0x2}, {0xc0000400b0?, 0x0, 0x0?})
    /Users/phil/work/zed/pkg/charm/path.go:11 +0x79
github.com/brimdata/zed/pkg/charm.(*Spec).ExecRoot(0xc0001da580?, {0xc000040090, 0x3, 0x3})
    /Users/phil/work/zed/pkg/charm/charm.go:63 +0x3f
main.main()
    /Users/phil/work/zed/cmd/zed/main.go:63 +0xf05

goroutine 37 [syscall]:
os/signal.signal_recv()
    /usr/local/opt/go@1.21/libexec/src/runtime/sigqueue.go:149 +0x25
os/signal.loop()
    /usr/local/opt/go@1.21/libexec/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
    /usr/local/opt/go@1.21/libexec/src/os/signal/signal.go:151 +0x1f

goroutine 23 [select]:
github.com/brimdata/zed/cli.signalContext.func1()
    /Users/phil/work/zed/cli/cli.go:70 +0x85
created by github.com/brimdata/zed/cli.signalContext in goroutine 1
    /Users/phil/work/zed/cli/cli.go:69 +0xd2

goroutine 25 [select]:
github.com/brimdata/zed/pkg/httpd.(*Server).serve(0xc0005a7540, {0x2b60a48, 0xc00015d590}, {0x2b5a7b0?, 0xc00059de60})
    /Users/phil/work/zed/pkg/httpd/server.go:75 +0x1a8
created by github.com/brimdata/zed/pkg/httpd.(*Server).Start in goroutine 1
    /Users/phil/work/zed/pkg/httpd/server.go:47 +0x3eb

goroutine 26 [semacquire]:
sync.runtime_Semacquire(0x2445b25?)
    /usr/local/opt/go@1.21/libexec/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x1009ecb?)
    /usr/local/opt/go@1.21/libexec/src/sync/waitgroup.go:116 +0x48
github.com/brimdata/zed/pkg/httpd.(*Server).Wait(...)
    /Users/phil/work/zed/pkg/httpd/server.go:59
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /Users/phil/.go/pkg/mod/golang.org/x/sync@v0.4.0/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
    /Users/phil/.go/pkg/mod/golang.org/x/sync@v0.4.0/errgroup/errgroup.go:72 +0x96

goroutine 27 [IO wait]:
internal/poll.runtime_pollWait(0x4ac991e0, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0006c0e80?, 0x8?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006c0e80)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc0006c0e80)
    /usr/local/opt/go@1.21/libexec/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00059de60)
    /usr/local/opt/go@1.21/libexec/src/net/tcpsock_posix.go:152 +0x1e
net.(*TCPListener).Accept(0xc00059de60)
    /usr/local/opt/go@1.21/libexec/src/net/tcpsock.go:315 +0x30
net/http.(*Server).Serve(0xc0006e40f0, {0x2b5a7b0, 0xc00059de60})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3061 +0x364
github.com/brimdata/zed/pkg/httpd.(*Server).serve.func1()
    /Users/phil/work/zed/pkg/httpd/server.go:67 +0x2c
created by github.com/brimdata/zed/pkg/httpd.(*Server).serve in goroutine 25
    /Users/phil/work/zed/pkg/httpd/server.go:66 +0x131

goroutine 841 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba500, {0xc00056a870?, 0x1019828?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 991 [runnable]:
github.com/brimdata/zed/zio/zngio.newBuffer(0xcd)
    /Users/phil/work/zed/zio/zngio/buffer.go:20 +0x59
github.com/brimdata/zed/zio/zngio.(*parser).readCompressedFrame(0xc00074e018, 0xc5?)
    /Users/phil/work/zed/zio/zngio/parser.go:196 +0x2c5
github.com/brimdata/zed/zio/zngio.(*parser).decodeTypes(0xc00074e018, 0x20?)
    /Users/phil/work/zed/zio/zngio/parser.go:61 +0x25
github.com/brimdata/zed/zio/zngio.(*parser).read(0xc00074e018)
    /Users/phil/work/zed/zio/zngio/parser.go:45 +0x185
github.com/brimdata/zed/zio/zngio.(*scanner).start.func1()
    /Users/phil/work/zed/zio/zngio/scanner.go:118 +0x85
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:104 +0x138

goroutine 838 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba380, {0x60?, 0xc000808480?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 847 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba880, {0xc0004f27e0?, 0x2b60a10?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 63 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 14
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 1039 [IO wait]:
internal/poll.runtime_pollWait(0x4ac990e8, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002782000?, 0xc0027840a1?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002782000, {0xc0027840a1, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc002782000, {0xc0027840a1?, 0x1019828?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc001018008, {0xc0027840a1?, 0x1017c0e?, 0xc000022278?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc002784090)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 1038
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 1061 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58700, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1040 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740000, {0x3?, 0x10dda19?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 839 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba400, {0x1?, 0x1?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 840 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba480, {0x100000c00067e000?, 0x4000002900000000?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 837 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba300, {0xc000a14908?, 0xc00051c2a0?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 978 [IO wait]:
internal/poll.runtime_pollWait(0x4ac98a20, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000fba080?, 0xc00082fc01?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fba080, {0xc00082fc01, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000fba080, {0xc00082fc01?, 0xc00249eea0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00007a048, {0xc00082fc01?, 0x0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc00082fbf0)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 801
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 1000 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724180, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1058 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58480, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 525 [select]:
github.com/brimdata/zed/zio/zngio.(*scanner).Pull(0xc00017a000, 0x40?)
    /Users/phil/work/zed/zio/zngio/scanner.go:80 +0x146
github.com/brimdata/zed/zbuf.(*pullerReader).Read(0xc000820140)
    /Users/phil/work/zed/zbuf/batch.go:190 +0x85
github.com/brimdata/zed/zio/zngio.(*Reader).Read(0xc0016ec2a0)
    /Users/phil/work/zed/zio/zngio/reader.go:106 +0x42
github.com/brimdata/zed/lake/journal.(*Store).load(0xc0004b9320, {0x2b60a10, 0xc00093c750})
    /Users/phil/work/zed/lake/journal/store.go:108 +0x497
github.com/brimdata/zed/lake/journal.(*Store).commit(0xc0004b9320, {0x2b60a10, 0xc00093c750}, 0xc0006b1238, {0xc001953218, 0x2, 0xc0005a02c0?})
    /Users/phil/work/zed/lake/journal/store.go:343 +0x13b
github.com/brimdata/zed/lake/journal.(*Store).Move(0xc0004b9320, {0x2b60a10, 0xc00093c750}, {0xc000015d70, 0x4}, {0x2b49880?, 0xc0001b0160})
    /Users/phil/work/zed/lake/journal/store.go:297 +0x190
github.com/brimdata/zed/lake/pools.(*Store).Rename(0xc0005b0060, {0x2b60a10, 0xc00093c750}, {0x13, 0x67, 0x8a, 0xb4, 0x92, 0x3d, 0x86, ...}, ...)
    /Users/phil/work/zed/lake/pools/store.go:91 +0xde
github.com/brimdata/zed/lake.(*Root).RenamePool(...)
    /Users/phil/work/zed/lake/root.go:314
github.com/brimdata/zed/service.handlePoolPut(0xc0005a9560, 0xc00098ff40, 0xc000af2950)
    /Users/phil/work/zed/service/handlers.go:277 +0x145
github.com/brimdata/zed/service.(*Core).authhandle.(*Core).handler.func2({0x2b5a900?, 0xc000815ca0?}, 0xc000833c20?)
    /Users/phil/work/zed/service/core.go:198 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005c49a0?, {0x2b5a900?, 0xc000815ca0?}, 0xc00116fd00?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x2b5a900, 0xc000815ca0}, 0xc00116fd00)
    /Users/phil/.go/pkg/mod/github.com/rs/cors@v1.8.0/cors.go:219 +0x17e
net/http.HandlerFunc.ServeHTTP(0xc0019534a8?, {0x2b5a900?, 0xc000815ca0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.panicCatchMiddleware.func6.1({0x2b5a900?, 0xc000815ca0?}, 0x10d120f?)
    /Users/phil/work/zed/service/middleware.go:92 +0x7a
net/http.HandlerFunc.ServeHTTP(0x2b5aa80?, {0x2b5a900?, 0xc000815ca0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.accessLogMiddleware.func5.1({0x2b5aa80, 0xc0005c00e0}, 0xc00116fd00)
    /Users/phil/work/zed/service/middleware.go:72 +0x8b4
net/http.HandlerFunc.ServeHTTP(0xc00116fc00?, {0x2b5aa80?, 0xc0005c00e0?}, 0x2b42470?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.requestIDMiddleware.func4.1({0x2b5aa80, 0xc0005c00e0}, 0xc00116fc00)
    /Users/phil/work/zed/service/middleware.go:28 +0x14b
net/http.HandlerFunc.ServeHTTP(0xc00116fb00?, {0x2b5aa80?, 0xc0005c00e0?}, 0xc001953aa8?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005dc240, {0x2b5aa80, 0xc0005c00e0}, 0xc00116fa00)
    /Users/phil/.go/pkg/mod/github.com/gorilla/mux@v1.7.5-0.20200711200521-98cb6bf42e08/mux.go:210 +0x1c5
github.com/brimdata/zed/service.(*Core).ServeHTTP(0xc0005a9560, {0x2b5aa80, 0xc0005c00e0}, 0xc0005c0001?)
    /Users/phil/work/zed/service/core.go:220 +0xa5
net/http.serverHandler.ServeHTTP({0x2b51be0?}, {0x2b5aa80?, 0xc0005c00e0?}, 0x6?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2943 +0x8e
net/http.(*conn).serve(0xc0005241b0, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2014 +0x5f4
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 1009 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724680, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 980 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782200, {0x0?, 0x0?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 929 [IO wait]:
internal/poll.runtime_pollWait(0x4ac98e00, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000fba000?, 0xc00104a0a1?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fba000, {0xc00104a0a1, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000fba000, {0xc00104a0a1?, 0xc0005b8660?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00007a038, {0xc00104a0a1?, 0x0?, 0xc000a16000?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc00104a090)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 928
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 844 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba680, {0x0?, 0x0?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1008 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724600, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1001 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724200, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 915 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 513
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 989 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782700, {0xc000afb7d0?, 0x1315d85?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 998 [IO wait]:
internal/poll.runtime_pollWait(0x4ac98d08, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000fba100?, 0xc00104a401?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fba100, {0xc00104a401, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000fba100, {0xc00104a401?, 0xc000f1ac60?, 0x2b51528?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00007a198, {0xc00104a401?, 0x0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc00104a3f0)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 997
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 592 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58380, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 843 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba600, {0x16?, 0x8?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 985 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782480, {0xc0009a27d0?, 0x1315d85?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 979 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782100, {0xc000f90480?, 0x2b60a10?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 842 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba580, {0xc000a9e240?, 0x3a1cad0?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 981 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782280, {0xc0005983f0?, 0x2b60a10?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 983 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782380, {0x0?, 0x0?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 984 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782400, {0x0?, 0x0?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 836 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba200, {0x2?, 0xc000571858?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 907 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 541
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 510 [running]:
    goroutine running on other thread; stack unavailable
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 884 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 695
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 845 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba780, {0x0?, 0x0?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1012 [IO wait]:
internal/poll.runtime_pollWait(0x4ac98ff0, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000fba180?, 0xc00104a3a1?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fba180, {0xc00104a3a1, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000fba180, {0xc00104a3a1?, 0x0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00007a1a8, {0xc00104a3a1?, 0x0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc00104a390)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 1011
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 988 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782680, {0xc000afc7d0?, 0x1315d85?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 967 [IO wait]:
internal/poll.runtime_pollWait(0x4ac98c10, 0x72)
    /usr/local/opt/go@1.21/libexec/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000740180?, 0xc000fc6281?, 0x0)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000740180, {0xc000fc6281, 0x1, 0x1})
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000740180, {0xc000fc6281?, 0xc0005ea4e0?, 0x0?})
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005b0018, {0xc000fc6281?, 0x0?, 0xc00100c420?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:185 +0x45
net/http.(*connReader).backgroundRead(0xc000fc6270)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:683 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 966
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:679 +0xba

goroutine 513 [select]:
github.com/brimdata/zed/zio/zngio.(*scanner).Pull(0xc0005a4140, 0x70?)
    /Users/phil/work/zed/zio/zngio/scanner.go:80 +0x146
github.com/brimdata/zed/zbuf.(*pullerReader).Read(0xc0005a0440)
    /Users/phil/work/zed/zbuf/batch.go:190 +0x85
github.com/brimdata/zed/zio/zngio.(*Reader).Read(0xc0009922a0)
    /Users/phil/work/zed/zio/zngio/reader.go:106 +0x42
github.com/brimdata/zed/lake/journal.(*Store).load(0xc0004b9320, {0x2b60a10, 0xc000fb81e0})
    /Users/phil/work/zed/lake/journal/store.go:108 +0x497
github.com/brimdata/zed/lake/journal.(*Store).commit(0xc0004b9320, {0x2b60a10, 0xc000fb81e0}, 0xc00117f238, {0xc000b9f218, 0x2, 0xc0005a02c0?})
    /Users/phil/work/zed/lake/journal/store.go:343 +0x13b
github.com/brimdata/zed/lake/journal.(*Store).Move(0xc0004b9320, {0x2b60a10, 0xc000fb81e0}, {0xc000015d70, 0x4}, {0x2b49880?, 0xc001166108})
    /Users/phil/work/zed/lake/journal/store.go:297 +0x190
github.com/brimdata/zed/lake/pools.(*Store).Rename(0xc0005b0060, {0x2b60a10, 0xc000fb81e0}, {0x13, 0x67, 0x8a, 0xb4, 0x92, 0x3d, 0x86, ...}, ...)
    /Users/phil/work/zed/lake/pools/store.go:91 +0xde
github.com/brimdata/zed/lake.(*Root).RenamePool(...)
    /Users/phil/work/zed/lake/root.go:314
github.com/brimdata/zed/service.handlePoolPut(0xc0005a9560, 0xc000aa0050, 0xc0019564a0)
    /Users/phil/work/zed/service/handlers.go:277 +0x145
github.com/brimdata/zed/service.(*Core).authhandle.(*Core).handler.func2({0x2b5a900?, 0xc0027883c0?}, 0xc0005c6200?)
    /Users/phil/work/zed/service/core.go:198 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005c49a0?, {0x2b5a900?, 0xc0027883c0?}, 0xc0024aa600?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x2b5a900, 0xc0027883c0}, 0xc0024aa600)
    /Users/phil/.go/pkg/mod/github.com/rs/cors@v1.8.0/cors.go:219 +0x17e
net/http.HandlerFunc.ServeHTTP(0xc000b9f4a8?, {0x2b5a900?, 0xc0027883c0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.panicCatchMiddleware.func6.1({0x2b5a900?, 0xc0027883c0?}, 0x10d120f?)
    /Users/phil/work/zed/service/middleware.go:92 +0x7a
net/http.HandlerFunc.ServeHTTP(0x2b5aa80?, {0x2b5a900?, 0xc0027883c0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.accessLogMiddleware.func5.1({0x2b5aa80, 0xc00012a2a0}, 0xc0024aa600)
    /Users/phil/work/zed/service/middleware.go:72 +0x8b4
net/http.HandlerFunc.ServeHTTP(0xc0024aa500?, {0x2b5aa80?, 0xc00012a2a0?}, 0x2b42470?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.requestIDMiddleware.func4.1({0x2b5aa80, 0xc00012a2a0}, 0xc0024aa500)
    /Users/phil/work/zed/service/middleware.go:28 +0x14b
net/http.HandlerFunc.ServeHTTP(0xc0024aa400?, {0x2b5aa80?, 0xc00012a2a0?}, 0xc000b9faa8?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005dc240, {0x2b5aa80, 0xc00012a2a0}, 0xc0024aa300)
    /Users/phil/.go/pkg/mod/github.com/gorilla/mux@v1.7.5-0.20200711200521-98cb6bf42e08/mux.go:210 +0x1c5
github.com/brimdata/zed/service.(*Core).ServeHTTP(0xc0005a9560, {0x2b5aa80, 0xc00012a2a0}, 0xc00012a201?)
    /Users/phil/work/zed/service/core.go:220 +0xa5
net/http.serverHandler.ServeHTTP({0x2b51be0?}, {0x2b5aa80?, 0xc00012a2a0?}, 0x6?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2943 +0x8e
net/http.(*conn).serve(0xc0004f2870, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2014 +0x5f4
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 1005 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724400, {0xc0004f26c0?, 0x2b60a10?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1006 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724480, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1002 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724280, {0xc0001bb920?, 0xc0001bb950?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1004 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724380, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1007 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724500, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 999 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724080, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1003 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724300, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1042 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc001724700, {0x0?, 0x0?}, 0xc001690180)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 818
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 591 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58100, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 982 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782300, {0xc00099a750?, 0x2b60a10?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 541 [select]:
github.com/brimdata/zed/zio/zngio.(*scanner).Pull(0xc00074e000, 0x60?)
    /Users/phil/work/zed/zio/zngio/scanner.go:80 +0x146
github.com/brimdata/zed/zbuf.(*pullerReader).Read(0xc00100e180)
    /Users/phil/work/zed/zbuf/batch.go:190 +0x85
github.com/brimdata/zed/zio/zngio.(*Reader).Read(0xc000a12900)
    /Users/phil/work/zed/zio/zngio/reader.go:106 +0x42
github.com/brimdata/zed/lake/journal.(*Store).load(0xc0004b9320, {0x2b60a10, 0xc000eef500})
    /Users/phil/work/zed/lake/journal/store.go:108 +0x497
github.com/brimdata/zed/lake/journal.(*Store).commit(0xc0004b9320, {0x2b60a10, 0xc000eef500}, 0xc000965238, {0xc00248b218, 0x2, 0xc0005a02c0?})
    /Users/phil/work/zed/lake/journal/store.go:343 +0x13b
github.com/brimdata/zed/lake/journal.(*Store).Move(0xc0004b9320, {0x2b60a10, 0xc000eef500}, {0xc000015d70, 0x4}, {0x2b49880?, 0xc0001b1c88})
    /Users/phil/work/zed/lake/journal/store.go:297 +0x190
github.com/brimdata/zed/lake/pools.(*Store).Rename(0xc0005b0060, {0x2b60a10, 0xc000eef500}, {0x13, 0x67, 0x8a, 0xb4, 0x92, 0x3d, 0x86, ...}, ...)
    /Users/phil/work/zed/lake/pools/store.go:91 +0xde
github.com/brimdata/zed/lake.(*Root).RenamePool(...)
    /Users/phil/work/zed/lake/root.go:314
github.com/brimdata/zed/service.handlePoolPut(0xc0005a9560, 0xc001958cd0, 0xc001595ed0)
    /Users/phil/work/zed/service/handlers.go:277 +0x145
github.com/brimdata/zed/service.(*Core).authhandle.(*Core).handler.func2({0x2b5a900?, 0xc0015f9e20?}, 0xc001828fc0?)
    /Users/phil/work/zed/service/core.go:198 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005c49a0?, {0x2b5a900?, 0xc0015f9e20?}, 0xc0015ff300?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x2b5a900, 0xc0015f9e20}, 0xc0015ff300)
    /Users/phil/.go/pkg/mod/github.com/rs/cors@v1.8.0/cors.go:219 +0x17e
net/http.HandlerFunc.ServeHTTP(0xc000c6b4a8?, {0x2b5a900?, 0xc0015f9e20?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.panicCatchMiddleware.func6.1({0x2b5a900?, 0xc0015f9e20?}, 0x10d120f?)
    /Users/phil/work/zed/service/middleware.go:92 +0x7a
net/http.HandlerFunc.ServeHTTP(0x2b5aa80?, {0x2b5a900?, 0xc0015f9e20?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.accessLogMiddleware.func5.1({0x2b5aa80, 0xc00012e2a0}, 0xc0015ff300)
    /Users/phil/work/zed/service/middleware.go:72 +0x8b4
net/http.HandlerFunc.ServeHTTP(0xc0015ff200?, {0x2b5aa80?, 0xc00012e2a0?}, 0x2b42470?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.requestIDMiddleware.func4.1({0x2b5aa80, 0xc00012e2a0}, 0xc0015ff200)
    /Users/phil/work/zed/service/middleware.go:28 +0x14b
net/http.HandlerFunc.ServeHTTP(0xc0015ff100?, {0x2b5aa80?, 0xc00012e2a0?}, 0xc000c6baa8?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005dc240, {0x2b5aa80, 0xc00012e2a0}, 0xc0015ff000)
    /Users/phil/.go/pkg/mod/github.com/gorilla/mux@v1.7.5-0.20200711200521-98cb6bf42e08/mux.go:210 +0x1c5
github.com/brimdata/zed/service.(*Core).ServeHTTP(0xc0005a9560, {0x2b5aa80, 0xc00012e2a0}, 0xc00012e201?)
    /Users/phil/work/zed/service/core.go:220 +0xa5
net/http.serverHandler.ServeHTTP({0x2b51be0?}, {0x2b5aa80?, 0xc00012e2a0?}, 0x6?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2943 +0x8e
net/http.(*conn).serve(0xc002786000, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2014 +0x5f4
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 986 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782500, {0x0?, 0x0?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 987 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782580, {0x0?, 0x0?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 593 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58400, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 846 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000fba800, {0x0?, 0x0?}, 0xc000f52540)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 510
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1059 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58600, {0xc00067e480?, 0x2b60a10?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1060 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58680, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 512 [runnable]:
net.(*conn).Write(0xc00007a048?, {0xc000461000?, 0x111?, 0x1000?})
    /usr/local/opt/go@1.21/libexec/src/net/net.go:193 +0x15e
net/http.checkConnErrorWriter.Write({0xc001048040?}, {0xc000461000?, 0xc0009b1b48?, 0x1316c45?})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3604 +0x2d
bufio.(*Writer).Flush(0xc00082c4c0)
    /usr/local/opt/go@1.21/libexec/src/bufio/bufio.go:642 +0x55
net/http.(*response).finishRequest(0xc001048000)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:1669 +0x76
net/http.(*conn).serve(0xc0004f27e0, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2020 +0x63a
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 1064 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58900, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1065 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58980, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1062 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58780, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1066 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58a00, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1067 [runnable]:
github.com/brimdata/zed.appendTypeValue({0xc000d84700, 0x41, 0x80}, {0x2b51528?, 0xc00084a8d0?}, 0xc000907c50)
    /Users/phil/work/zed/type.go:497 +0xc85
github.com/brimdata/zed.appendTypeValue({0xc00158dac0, 0x32, 0x40}, {0x2b51500?, 0xc00084a9c0?}, 0xc000907c50)
    /Users/phil/work/zed/type.go:513 +0xb88
github.com/brimdata/zed.appendTypeValue({0xc00158dac0, 0xc002493130?, 0x40}, {0x2b51528?, 0xc00084aa80?}, 0xc000907c50)
    /Users/phil/work/zed/type.go:526 +0x83a
github.com/brimdata/zed.appendTypeValue({0xc00158dac0, 0x0, 0x40}, {0x2b51500?, 0xc00084ab10?}, 0xc000907c50)
    /Users/phil/work/zed/type.go:513 +0xb88
github.com/brimdata/zed.AppendTypeValue(...)
    /Users/phil/work/zed/type.go:494
github.com/brimdata/zed.(*Context).LookupTypeNamed(0xc000f8a370, {0xc002493130, 0xc}, {0x2b51528?, 0xc00084aa80})
    /Users/phil/work/zed/context.go:250 +0x189
github.com/brimdata/zed/zio/zngio.(*Decoder).readTypeName(0xc000ab8000, 0xc00105e005?)
    /Users/phil/work/zed/zio/zngio/types.go:424 +0x8d
github.com/brimdata/zed/zio/zngio.(*Decoder).decode(0xc000907da8?, 0xc0015f8000)
    /Users/phil/work/zed/zio/zngio/types.go:255 +0xac
github.com/brimdata/zed/zio/zngio.(*parser).decodeTypes(0xc0005a4158, 0x20?)
    /Users/phil/work/zed/zio/zngio/parser.go:68 +0x5d
github.com/brimdata/zed/zio/zngio.(*parser).read(0xc0005a4158)
    /Users/phil/work/zed/zio/zngio/parser.go:45 +0x185
github.com/brimdata/zed/zio/zngio.(*scanner).start.func1()
    /Users/phil/work/zed/zio/zngio/scanner.go:118 +0x85
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:104 +0x138

goroutine 1063 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000f58800, {0x0?, 0x0?}, 0xc000992360)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 513
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 636 [runnable]:
syscall.syscall(0xc000975988?, 0x106c485?, 0x3aa7d30?, 0x4ac98b18?)
    /usr/local/opt/go@1.21/libexec/src/runtime/sys_darwin.go:23 +0x70
syscall.Close(0xc0009759d8?)
    /usr/local/opt/go@1.21/libexec/src/syscall/zsyscall_darwin_amd64.go:509 +0x29
internal/poll.(*SysFile).destroy(...)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unixjs.go:24
internal/poll.(*FD).destroy(0xc0001dbc80)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:81 +0x51
internal/poll.(*FD).decref(0x4ac98b18?)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_mutex.go:213 +0x53
internal/poll.(*FD).Close(0xc0001dbc80)
    /usr/local/opt/go@1.21/libexec/src/internal/poll/fd_unix.go:104 +0x45
net.(*netFD).Close(0xc0001dbc80)
    /usr/local/opt/go@1.21/libexec/src/net/fd_posix.go:37 +0x32
net.(*conn).Close(0xc00105c0a8)
    /usr/local/opt/go@1.21/libexec/src/net/net.go:209 +0x36
net/http.(*conn).close(0xc00067e360)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:1751 +0x2b
net/http.(*conn).serve.func1()
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:1884 +0x1cc
net/http.(*conn).serve(0xc00067e360, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2050 +0x76e
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb

goroutine 788 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 512
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 879 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 636
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 886 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 525
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 820 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 510
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 858 [sleep]:
time.Sleep(0x2540be400)
    /usr/local/opt/go@1.21/libexec/src/runtime/time.go:195 +0x125
github.com/brimdata/zed/service.(*Core).newQueryStatus.func1()
    /Users/phil/work/zed/service/core.go:249 +0x3c
created by github.com/brimdata/zed/service.(*queryStatus).Done in goroutine 818
    /Users/phil/work/zed/service/core.go:276 +0x33

goroutine 990 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc002782780, {0xc000afcfd0?, 0x1315d85?}, 0xc000a12960)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 541
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1041 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740080, {0xc000571fd0?, 0x1315d85?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1074 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740100, {0xc00082e5d0?, 0xc000f94c60?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1075 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740200, {0xc0005427d0?, 0x1315d85?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1076 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740300, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1077 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740400, {0xc0004f2900?, 0x2b60a10?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1078 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740500, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1079 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740580, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1080 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740600, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1081 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740700, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1082 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740780, {0x0?, 0x0?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1083 [select]:
github.com/brimdata/zed/zio/zngio.(*worker).run(0xc000740800, {0xc000f90d80?, 0x2b60a10?}, 0xc0016ec300)
    /Users/phil/work/zed/zio/zngio/scanner.go:222 +0x128
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:102 +0x3a

goroutine 1084 [runnable]:
github.com/brimdata/zed/zio/zngio.(*Decoder).readTypeRecord(0xc000a18180, 0xc001206005?)
    /Users/phil/work/zed/zio/zngio/types.go:281 +0x184
github.com/brimdata/zed/zio/zngio.(*Decoder).decode(0xc000901da8?, 0xc00182a1c0)
    /Users/phil/work/zed/zio/zngio/types.go:243 +0x75
github.com/brimdata/zed/zio/zngio.(*parser).decodeTypes(0xc00017a018, 0x20?)
    /Users/phil/work/zed/zio/zngio/parser.go:68 +0x5d
github.com/brimdata/zed/zio/zngio.(*parser).read(0xc00017a018)
    /Users/phil/work/zed/zio/zngio/parser.go:45 +0x185
github.com/brimdata/zed/zio/zngio.(*scanner).start.func1()
    /Users/phil/work/zed/zio/zngio/scanner.go:118 +0x85
created by github.com/brimdata/zed/zio/zngio.(*scanner).start in goroutine 525
    /Users/phil/work/zed/zio/zngio/scanner.go:104 +0x138

goroutine 510 [running]:
github.com/brimdata/zed/zson.(*binder).enter(...)
    /Users/phil/work/zed/zson/marshal.go:1082
github.com/brimdata/zed/zson.(*binder).enterBinding(0xc000594958, {{0x2850331?, 0x269c260?}, {0x27930c0?, 0x3a766e0?}})
    /Users/phil/work/zed/zson/marshal.go:1116 +0x165
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).NamedBindings(...)
    /Users/phil/work/zed/zson/marshal.go:696
github.com/brimdata/zed/lake/pools.(*Config).UnmarshalZNG(0xc002792960, 0xc000594948, {0x0?, 0x0?})
    /Users/phil/work/zed/lake/pools/config.go:96 +0xaf
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc002792960?, 0xc001660f60?}, {0x27a63c0?, 0xc002792960?, 0x7fff6?})
    /Users/phil/work/zed/zson/marshal.go:712 +0x2643
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000af2120?, 0xc001660f60?}, {0x263b120?, 0xc000af2120?, 0x263b120?})
    /Users/phil/work/zed/zson/marshal.go:791 +0x1f39
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeRecord(0x2b51500?, {0x100000c00099ab40, 0x4000002900000001}, {0x269cbc0?, 0xc000af2120?, 0x0?})
    /Users/phil/work/zed/zson/marshal.go:978 +0x7d7
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000af2120?, 0xc001661920?}, {0x269cbc0?, 0xc000af2120?, 0x7fff7?})
    /Users/phil/work/zed/zson/marshal.go:759 +0x252e
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).decodeAny(0xc000594948, {0xc000af2110?, 0x0?}, {0x2532780?, 0xc000af2110?, 0x10?})
    /Users/phil/work/zed/zson/marshal.go:791 +0x1f39
github.com/brimdata/zed/zson.(*UnmarshalZNGContext).Unmarshal(0xc000f520c0?, {0xc0006388d0?, 0xc00278b770?}, {0x2532780?, 0xc000af2110?})
    /Users/phil/work/zed/zson/marshal.go:674 +0x6b
github.com/brimdata/zed/lake/journal.(*Store).load(0xc0004b9320, {0x2b60a10, 0xc00081dda0})
    /Users/phil/work/zed/lake/journal/store.go:129 +0x50b
github.com/brimdata/zed/lake/journal.(*Store).commit(0xc0004b9320, {0x2b60a10, 0xc00081dda0}, 0xc0008d9238, {0xc00165f218, 0x2, 0xc0005a02c0?})
    /Users/phil/work/zed/lake/journal/store.go:343 +0x13b
github.com/brimdata/zed/lake/journal.(*Store).Move(0xc0004b9320, {0x2b60a10, 0xc00081dda0}, {0xc000015d70, 0x4}, {0x2b49880?, 0xc0005df970})
    /Users/phil/work/zed/lake/journal/store.go:297 +0x190
github.com/brimdata/zed/lake/pools.(*Store).Rename(0xc0005b0060, {0x2b60a10, 0xc00081dda0}, {0x13, 0x67, 0x8a, 0xb4, 0x92, 0x3d, 0x86, ...}, ...)
    /Users/phil/work/zed/lake/pools/store.go:91 +0xde
github.com/brimdata/zed/lake.(*Root).RenamePool(...)
    /Users/phil/work/zed/lake/root.go:314
github.com/brimdata/zed/service.handlePoolPut(0xc0005a9560, 0xc000f0fd10, 0xc000f85500)
    /Users/phil/work/zed/service/handlers.go:277 +0x145
github.com/brimdata/zed/service.(*Core).authhandle.(*Core).handler.func2({0x2b5a900?, 0xc00059daa0?}, 0xc000f7ca40?)
    /Users/phil/work/zed/service/core.go:198 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005c49a0?, {0x2b5a900?, 0xc00059daa0?}, 0xc000ef5700?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/rs/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x2b5a900, 0xc00059daa0}, 0xc000ef5700)
    /Users/phil/.go/pkg/mod/github.com/rs/cors@v1.8.0/cors.go:219 +0x17e
net/http.HandlerFunc.ServeHTTP(0xc00165f4a8?, {0x2b5a900?, 0xc00059daa0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.panicCatchMiddleware.func6.1({0x2b5a900?, 0xc00059daa0?}, 0x10d120f?)
    /Users/phil/work/zed/service/middleware.go:92 +0x7a
net/http.HandlerFunc.ServeHTTP(0x2b5aa80?, {0x2b5a900?, 0xc00059daa0?}, 0x0?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.accessLogMiddleware.func5.1({0x2b5aa80, 0xc000144000}, 0xc000ef5700)
    /Users/phil/work/zed/service/middleware.go:72 +0x8b4
net/http.HandlerFunc.ServeHTTP(0xc000ef5600?, {0x2b5aa80?, 0xc000144000?}, 0x2b42470?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/brimdata/zed/service.NewCore.requestIDMiddleware.func4.1({0x2b5aa80, 0xc000144000}, 0xc000ef5600)
    /Users/phil/work/zed/service/middleware.go:28 +0x14b
net/http.HandlerFunc.ServeHTTP(0xc000ef5500?, {0x2b5aa80?, 0xc000144000?}, 0xc00165faa8?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2141 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0005dc240, {0x2b5aa80, 0xc000144000}, 0xc000ef5400)
    /Users/phil/.go/pkg/mod/github.com/gorilla/mux@v1.7.5-0.20200711200521-98cb6bf42e08/mux.go:210 +0x1c5
github.com/brimdata/zed/service.(*Core).ServeHTTP(0xc0005a9560, {0x2b5aa80, 0xc000144000}, 0xc000144001?)
    /Users/phil/work/zed/service/core.go:220 +0xa5
net/http.serverHandler.ServeHTTP({0x2b51be0?}, {0x2b5aa80?, 0xc000144000?}, 0x6?)
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2943 +0x8e
net/http.(*conn).serve(0xc0004f25a0, {0x2b60a10, 0xc0006e05d0})
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:2014 +0x5f4
created by net/http.(*Server).Serve in goroutine 27
    /usr/local/opt/go@1.21/libexec/src/net/http/server.go:3091 +0x5cb
philrz commented 3 weeks ago

CI recently had a test failure that may share a root cause with other failure described above, since the lake/journal/journal_test.go test in question purports to "TestJournalConcurrent" and I see similar themes in both stack traces. I also see an open issue #2546 that implies there may be a known problem with concurrent use of the journal.

The full stack trace from this recent failure:

panic: test timed out after 10m0s
running tests:
    TestJournalConcurrent (10m0s)

goroutine 61 [running]:
testing.(*M).startAlarm.func1()
    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:2259 +0x3b9
created by time.goFunc
    /opt/hostedtoolcache/go/1.21.13/x64/src/time/sleep.go:176 +0x2d

    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/internal/testdeps/deps.go:78 +0x2a
internal/testlog.Open({0xc000181080?, 0x5f92a6?})
    /opt/hostedtoolcache/go/1.21.13/x64/src/internal/testlog/log.go:60 +0x8b
os.OpenFile({0xc000181080, 0x2e}, 0x2c1, 0x8b90a0?)
    /opt/hostedtoolcache/go/1.21.13/x64/src/os/file.go:333 +0x26
github.com/brimdata/zed/pkg/storage.(*FileSystem).PutIfNotExists(0xc000249f50, {0x2?, 0xc000096610?}, 0xc000184480, {0xc000096600, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/file.go:51 +0x73
github.com/brimdata/zed/pkg/storage.(*Router).PutIfNotExists(0xc000249f80?, {0xae5020, 0xdd4d40}, 0x2?, {0xc000096600, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/router.go:83 +0x72
github.com/brimdata/zed/lake/journal.(*Queue).CommitAt(0xc000249f80, {0xae5020, 0xdd4d40}, 0xc000052108?, {0xc000096600, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:112 +0xa2
github.com/brimdata/zed/lake/journal.(*Queue).Commit(0xc000249f80, {0xae5020, 0xdd4d40}, {0xc000096600, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:100 +0x96
github.com/brimdata/zed/lake/journal.TestJournalConcurrent.func1(0x0?)
    /home/runner/work/zed/zed/lake/journal/journal_test.go:29 +0x85
created by github.com/brimdata/zed/lake/journal.TestJournalConcurrent in goroutine 6
    /home/runner/work/zed/zed/lake/journal/journal_test.go:27 +0x65

goroutine 47 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x897ee0?, 0xf2?, 0x2f[310](https://github.com/brimdata/zed/actions/runs/10865226064/job/30151453342#step:8:311)00000897ea0?)
    /opt/hostedtoolcache/go/1.21.13/x64/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xda4e20)
    /opt/hostedtoolcache/go/1.21.13/x64/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
    /opt/hostedtoolcache/go/1.21.13/x64/src/sync/mutex.go:90
testing/internal/testdeps.(*testLog).add(0xda4e20, {0x95fcd7, 0x4}, {0xc0000daae0, 0x2e})
    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/internal/testdeps/deps.go:95 +0x86
testing/internal/testdeps.(*testLog).Open(0xc0000daae0?, {0xc0000daae0?, 0xc000343d40?})
    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/internal/testdeps/deps.go:78 +0x2a
internal/testlog.Open({0xc0000daae0?, 0x5f92a6?})
    /opt/hostedtoolcache/go/1.21.13/x64/src/internal/testlog/log.go:60 +0x8b
os.OpenFile({0xc0000daae0, 0x2e}, 0x2c1, 0x8b90a0?)
    /opt/hostedtoolcache/go/1.21.13/x64/src/os/file.go:333 +0x26
github.com/brimdata/zed/pkg/storage.(*FileSystem).PutIfNotExists(0xc000249f50, {0x2?, 0xc0002c3f80?}, 0xc0000d8f30, {0xc000097030, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/file.go:51 +0x73
github.com/brimdata/zed/pkg/storage.(*Router).PutIfNotExists(0xc000249f80?, {0xae5020, 0xdd4d40}, 0x2?, {0xc000097030, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/router.go:83 +0x72
github.com/brimdata/zed/lake/journal.(*Queue).CommitAt(0xc000249f80, {0xae5020, 0xdd4d40}, 0xc000052108?, {0xc000097030, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:112 +0xa2
github.com/brimdata/zed/lake/journal.(*Queue).Commit(0xc000249f80, {0xae5020, 0xdd4d40}, {0xc000097030, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:100 +0x96
github.com/brimdata/zed/lake/journal.TestJournalConcurrent.func1(0x0?)
    /home/runner/work/zed/zed/lake/journal/journal_test.go:29 +0x85
created by github.com/brimdata/zed/lake/journal.TestJournalConcurrent in goroutine 6
    /home/runner/work/zed/zed/lake/journal/journal_test.go:27 +0x65

goroutine 52 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x897ee0?, 0xf2?, 0x2f31000000897ea0?)
    /opt/hostedtoolcache/go/1.21.13/x64/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xda4e20)
    /opt/hostedtoolcache/go/1.21.13/x64/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
    /opt/hostedtoolcache/go/1.21.13/x64/src/sync/mutex.go:90
testing/internal/testdeps.(*testLog).add(0xda4e20, {0x95fcd7, 0x4}, {0xc0000f0480, 0x2e})
    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/internal/testdeps/deps.go:95 +0x86
testing/internal/testdeps.(*testLog).Open(0xc0000f0480?, {0xc0000f0480?, 0xc000356d40?})
    /opt/hostedtoolcache/go/1.21.13/x64/src/testing/internal/testdeps/deps.go:78 +0x2a
internal/testlog.Open({0xc0000f0480?, 0x5f92a6?})
    /opt/hostedtoolcache/go/1.21.13/x64/src/internal/testlog/log.go:60 +0x8b
os.OpenFile({0xc0000f0480, 0x2e}, 0x2c1, 0x8b90a0?)
    /opt/hostedtoolcache/go/1.21.13/x64/src/os/file.go:[333](https://github.com/brimdata/zed/actions/runs/10865226064/job/30151453342#step:8:334) +0x26
github.com/brimdata/zed/pkg/storage.(*FileSystem).PutIfNotExists(0xc000249f50, {0x2?, 0xc0000e42b0?}, 0xc0000f42d0, {0xc0000e42a0, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/file.go:51 +0x73
github.com/brimdata/zed/pkg/storage.(*Router).PutIfNotExists(0xc000249f80?, {0xae5020, 0xdd4d40}, 0x2?, {0xc0000e42a0, 0xc, 0xc})
    /home/runner/work/zed/zed/pkg/storage/router.go:83 +0x72
github.com/brimdata/zed/lake/journal.(*Queue).CommitAt(0xc000249f80, {0xae5020, 0xdd4d40}, 0xc000052108?, {0xc0000e42a0, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:112 +0xa2
github.com/brimdata/zed/lake/journal.(*Queue).Commit(0xc000249f80, {0xae5020, 0xdd4d40}, {0xc0000e42a0, 0xc, 0xc})
    /home/runner/work/zed/zed/lake/journal/queue.go:100 +0x96
github.com/brimdata/zed/lake/journal.TestJournalConcurrent.func1(0x0?)
    /home/runner/work/zed/zed/lake/journal/journal_test.go:29 +0x85
created by github.com/brimdata/zed/lake/journal.TestJournalConcurrent in goroutine 6
    /home/runner/work/zed/zed/lake/journal/journal_test.go:27 +0x65
FAIL    github.com/brimdata/zed/lake/journal    600.020s
philrz commented 3 weeks ago

Verified in Zed commit 4a1ebf5.

I can now run the repro script shown above several times without triggering a crash.

$ zed -version
Version: v1.17.0-64-g4a1ebf5b

$ ./repro.sh 
pool created: foo1 2mAe99RSWnxdKJ6CeDLxqsQJzTQ
pool created: foo4 2mAe96i4KMljn9QVme5dCF51R0p
pool created: foo7 2mAe9BPOIguWTEeqwABDXEJXc00
pool created: foo2 2mAe96OIGDPkmZjWOkWbF7jHCDk
pool created: foo3 2mAe98KOHVnkrrUGsfqJACrSycm
pool created: foo8 2mAe98JzimpefmAUb4WdpOXydyp
pool created: foo5 2mAe9Az6uXilePVWeGPRHvj4xhQ
pool created: foo6 2mAe97DsnIrGEBLTkCiPb5llCdX
pool 2mAe99RSWnxdKJ6CeDLxqsQJzTQ renamed from foo1 to bar1
pool 2mAe96OIGDPkmZjWOkWbF7jHCDk renamed from foo2 to bar2
pool 2mAe98KOHVnkrrUGsfqJACrSycm renamed from foo3 to bar3
pool 2mAe96i4KMljn9QVme5dCF51R0p renamed from foo4 to bar4
pool 2mAe9Az6uXilePVWeGPRHvj4xhQ renamed from foo5 to bar5
pool 2mAe97DsnIrGEBLTkCiPb5llCdX renamed from foo6 to bar6
pool 2mAe9BPOIguWTEeqwABDXEJXc00 renamed from foo7 to bar7
pool 2mAe98JzimpefmAUb4WdpOXydyp renamed from foo8 to bar8
pool deleted: bar1
pool deleted: bar2
pool deleted: bar3
pool deleted: bar4
pool deleted: bar5
pool deleted: bar6
pool deleted: bar7
pool deleted: bar8

$ ./repro.sh 
pool created: foo8 2mAe9rN1rrZDeQwIQHkfL7atrZz
pool created: foo2 2mAe9pzFqfzt8xkJv8HGN8TN6sT
pool created: foo5 2mAe9svzRMolIIRx2TMVhilChYV
pool created: foo4 2mAe9xA9ZK0I1XM72NXiaeOrwPt
pool created: foo6 2mAe9uvvewyhKCniSVbdnW6kxeT
pool created: foo7 2mAe9vQK4CuL31k8xU0v7mINt2c
pool created: foo3 2mAe9wzTOIK6yCyO1A0gCHZbbiB
pool created: foo1 2mAe9sxvSHQBCIuk1irBK5FjV6C
pool 2mAe9wzTOIK6yCyO1A0gCHZbbiB renamed from foo3 to bar3
pool 2mAe9uvvewyhKCniSVbdnW6kxeT renamed from foo6 to bar6
pool 2mAe9rN1rrZDeQwIQHkfL7atrZz renamed from foo8 to bar8
pool 2mAe9vQK4CuL31k8xU0v7mINt2c renamed from foo7 to bar7
pool 2mAe9pzFqfzt8xkJv8HGN8TN6sT renamed from foo2 to bar2
pool 2mAe9svzRMolIIRx2TMVhilChYV renamed from foo5 to bar5
pool 2mAe9xA9ZK0I1XM72NXiaeOrwPt renamed from foo4 to bar4
pool 2mAe9sxvSHQBCIuk1irBK5FjV6C renamed from foo1 to bar1
pool deleted: bar1
pool deleted: bar2
pool deleted: bar3
pool deleted: bar4
pool deleted: bar5
pool deleted: bar6
pool deleted: bar7
pool deleted: bar8

$ ./repro.sh 
pool created: foo1 2mAeADJOYZUZesyvAtqCvSZVBUD
pool created: foo2 2mAeAJXr63YDMiy0jy2E2HXmRZX
pool created: foo8 2mAeAImuLCyyW0WN8xXBX5Ulc5S
pool created: foo5 2mAeAI9pBiR2SfyzoaqSw1pQQ6K
pool created: foo3 2mAeAH1j5eez5QsyfdenM6QZxWY
pool created: foo6 2mAeAGSz6NWPkKL615MPPY2M04r
pool created: foo7 2mAeAKYjQQY9sPimQMR9uYYKwG3
pool created: foo4 2mAeAHAXZTkzTjEP9lNJBcS2vCh
pool 2mAeAH1j5eez5QsyfdenM6QZxWY renamed from foo3 to bar3
pool 2mAeAGSz6NWPkKL615MPPY2M04r renamed from foo6 to bar6
pool 2mAeAI9pBiR2SfyzoaqSw1pQQ6K renamed from foo5 to bar5
pool 2mAeAJXr63YDMiy0jy2E2HXmRZX renamed from foo2 to bar2
pool 2mAeAHAXZTkzTjEP9lNJBcS2vCh renamed from foo4 to bar4
pool 2mAeAImuLCyyW0WN8xXBX5Ulc5S renamed from foo8 to bar8
pool 2mAeAKYjQQY9sPimQMR9uYYKwG3 renamed from foo7 to bar7
pool 2mAeADJOYZUZesyvAtqCvSZVBUD renamed from foo1 to bar1
pool deleted: bar1
pool deleted: bar2
pool deleted: bar3
pool deleted: bar4
pool deleted: bar5
pool deleted: bar6
pool deleted: bar7
pool deleted: bar8

Thanks @nwt!

philrz commented 2 weeks ago

Interestingly, in the time since this merged, we've had another failure in CI of this same src/js/api/pools/pools-api.test.ts. There's no longer a Zed lake service crash so it does seem like we're still seeing the desired improvement from the fix. But it would indicate there's still something to look at closer here.

The failure text from the CI job:

FAIL apps/zui/src/js/api/pools/pools-api.test.ts (6.68 s)
  ● rename pool

    FetchError: request to http://localhost:49427/pool/0x137f6e0f83bfc054f398861eb6e9faf0e0cd5c0f failed, reason:

      at ClientRequestOverride.<anonymous> (../../node_modules/node-fetch/lib/index.js:1501:11)
      at ClientRequest.<anonymous> (../../node_modules/@mswjs/interceptors/src/interceptors/ClientRequest/createClientRequestOverride.ts:309:14)

  ● rename pool

    FetchError: request to http://localhost:49427/pool/0x137f6e0f83bfc054f398861eb6e9faf0e0cd5c0f failed, reason:

       6 | ): Promise<Awaited<ReturnType<Operations[K]>>> {
       7 |   return global.zui.invoke(name, ...args).catch((error) => {
    >  8 |     throw new Error(
         |           ^
       9 |       error
      10 |         .toString()
      11 |         // Here we are stripping the error prefix that electron tacks on.

      at src/core/invoke.ts:8:11
      at PoolsApi.update (src/js/api/pools/pools-api.ts:57:5)
      at Object.<anonymous> (src/js/api/pools/pools-api.test.ts:18:3)