search

JuliaHEP / UnROOT.jl

Native Julia I/O package to work with CERN ROOT files objects (TTree and RNTuple)
https://juliahep.github.io/UnROOT.jl/
MIT License
97 stars 17 forks source link

use blocking @ccall for xrootdgo #311

Closed Moelf closed 6 months ago

Moelf commented 6 months ago

fix #310

This MAY make xrootd much slower but at least it's not wrong

codecov[bot] commented 6 months ago

Codecov Report

Attention: Patch coverage is 0% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 86.39%. Comparing base (1579c30) to head (469fbb9).

Files Patch % Lines
src/streamsource.jl 0.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #311 +/- ## ======================================= Coverage 86.39% 86.39% ======================================= Files 18 18 Lines 2447 2447 ======================================= Hits 2114 2114 Misses 333 333 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

peremato commented 6 months ago

With this change the reproducer works but my original program fails catastrophically. I would need to improve the reproducer. Perhaps has to do with the file itself and what actually is stored there. Here is the crash:

signal 16 received but handler not on signal stack
fatal error: non-Go code set up signal handler without SA_ONSTACK flag

runtime stack:
runtime.throw({0x14cdae3ca?, 0x14000081680?})
        /opt/x86_64-linux-musl/go/src/runtime/panic.go:1047 +0x40 fp=0x140001d5260 sp=0x140001d5230 pc=0x14cc6c350
runtime.sigNotOnStack(0x10)
        /opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:1020 +0x5c fp=0x140001d5280 sp=0x140001d5260 pc=0x14cc82b0c
runtime.adjustSignalStack(0x10, 0x1400004fb00, 0x140001d5328)
        /opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:581 +0x250 fp=0x140001d52f0 sp=0x140001d5280 pc=0x14cc81e10
runtime.sigtrampgo(0x10, 0x140001d5490, 0x140001d54f8)
        /opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:469 +0x174 fp=0x140001d5370 sp=0x140001d52f0 pc=0x14cc81a64
runtime.sigtrampgo(0x10, 0x140001d5490, 0x140001d54f8)
        <autogenerated>:1 +0x1c fp=0x140001d53a0 sp=0x140001d5370 pc=0x14cc9c78c
runtime.sigtramp()
        /opt/x86_64-linux-musl/go/src/runtime/sys_darwin_arm64.s:227 +0x4c fp=0x140001d5460 sp=0x140001d53a0 pc=0x14cc9bd3c

goroutine 3219 [syscall, locked to thread]:
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400005cfe0 sp=0x1400005cfe0 pc=0x14cc9af94

goroutine 36 [select]:
runtime.gopark(0x1400004c7a8?, 0x2?, 0x96?, 0xc7?, 0x1400004c79c?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004c640 sp=0x1400004c620 pc=0x14cc6ec94
runtime.selectgo(0x1400004c7a8, 0x1400004c798, 0x0?, 0x1, 0x100010000?, 0x1)
        /opt/x86_64-linux-musl/go/src/runtime/select.go:328 +0x6b4 fp=0x1400004c750 sp=0x1400004c640 pc=0x14cc7eba4
go-hep.org/x/hep/xrootd/internal/mux.New.func1()
        /workspace/srcdir/xrootd/internal/mux/mux.go:129 +0x88 fp=0x1400004c7d0 sp=0x1400004c750 pc=0x14cd990b8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004c7d0 sp=0x1400004c7d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd/internal/mux.New
        /workspace/srcdir/xrootd/internal/mux/mux.go:126 +0x114

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004afa0 sp=0x1400004af80 pc=0x14cc6ec94
runtime.goparkunlock(...)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.forcegchelper()
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:302 +0xb4 fp=0x1400004afd0 sp=0x1400004afa0 pc=0x14cc6eb24
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004afd0 sp=0x1400004afd0 pc=0x14cc9af94
created by runtime.init.6
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:290 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004b770 sp=0x1400004b750 pc=0x14cc6ec94
runtime.goparkunlock(...)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/mgcsweep.go:297 +0x10c fp=0x1400004b7b0 sp=0x1400004b770 pc=0x14cc5c15c
runtime.gcenable.func1()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:178 +0x28 fp=0x1400004b7d0 sp=0x1400004b7b0 pc=0x14cc507b8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004b7d0 sp=0x1400004b7d0 pc=0x14cc9af94
created by runtime.gcenable
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:178 +0x70

goroutine 4 [runnable]:
runtime.gopark(0x5e6998?, 0x175fbaa?, 0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004bf50 sp=0x1400004bf30 pc=0x14cc6ec94
runtime.goparkunlock(...)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x14cf42c60)
        /opt/x86_64-linux-musl/go/src/runtime/mgcscavenge.go:389 +0x5c fp=0x1400004bf80 sp=0x1400004bf50 pc=0x14cc5a0bc
runtime.bgscavenge(0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/mgcscavenge.go:622 +0xac fp=0x1400004bfb0 sp=0x1400004bf80 pc=0x14cc5a6bc
runtime.gcenable.func2()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:179 +0x28 fp=0x1400004bfd0 sp=0x1400004bfb0 pc=0x14cc50758
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004bfd0 sp=0x1400004bfd0 pc=0x14cc9af94
created by runtime.gcenable
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:179 +0xb4

goroutine 5 [finalizer wait]:
runtime.gopark(0x60000000000000?, 0x0?, 0x8?, 0x81?, 0x2000000000?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004a580 sp=0x1400004a560 pc=0x14cc6ec94
runtime.goparkunlock(...)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.runfinq()
        /opt/x86_64-linux-musl/go/src/runtime/mfinal.go:180 +0x128 fp=0x1400004a7d0 sp=0x1400004a580 pc=0x14cc4f9d8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004a7d0 sp=0x1400004a7d0 pc=0x14cc9af94
created by runtime.createfing
        /opt/x86_64-linux-musl/go/src/runtime/mfinal.go:157 +0x94

goroutine 18 [running, locked to thread]:
        goroutine running on other thread; stack unavailable

goroutine 7 [select]:
runtime.gopark(0x1400004cfa8?, 0x2?, 0x96?, 0xcf?, 0x1400004cf9c?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004ce40 sp=0x1400004ce20 pc=0x14cc6ec94
runtime.selectgo(0x1400004cfa8, 0x1400004cf98, 0x0?, 0x1, 0x0?, 0x1)
        /opt/x86_64-linux-musl/go/src/runtime/select.go:328 +0x6b4 fp=0x1400004cf50 sp=0x1400004ce40 pc=0x14cc7eba4
go-hep.org/x/hep/xrootd/internal/mux.New.func1()
        /workspace/srcdir/xrootd/internal/mux/mux.go:129 +0x88 fp=0x1400004cfd0 sp=0x1400004cf50 pc=0x14cd990b8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004cfd0 sp=0x1400004cfd0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd/internal/mux.New
        /workspace/srcdir/xrootd/internal/mux/mux.go:126 +0x114

goroutine 37 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000058c70 sp=0x14000058c50 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x14000058cb0 sp=0x14000058c70 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4f08, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x14000058ce0 sp=0x14000058cb0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x14000192000?, 0x140000164d0?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x14000058d10 sp=0x14000058ce0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000192000, {0x140000164d0, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x14000058d90 sp=0x14000058d10 pc=0x14ccda678
net.(*netFD).Read(0x14000192000, {0x140000164d0?, 0x14000198138?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x14000058de0 sp=0x14000058d90 pc=0x14cd05048
net.(*conn).Read(0x1400019c000, {0x140000164d0?, 0x140001a6018?, 0x14000058e8e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x14000058e30 sp=0x14000058de0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x1400019c000}, {0x140000164d0, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x14000058e80 sp=0x14000058e30 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x1400019c000}, {0x140000164d0, 0x8, 0x8}, 0x14000058f40)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x14000058ef0 sp=0x14000058e80 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x140001a2000)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x14000058fb0 sp=0x14000058ef0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSession.func1()
        /workspace/srcdir/xrootd/session.go:104 +0x28 fp=0x14000058fd0 sp=0x14000058fb0 pc=0x14cd9d0c8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000058fd0 sp=0x14000058fd0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSession
        /workspace/srcdir/xrootd/session.go:104 +0x264

goroutine 8 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000046470 sp=0x14000046450 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x140000464b0 sp=0x14000046470 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4e18, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x140000464e0 sp=0x140000464b0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x140000b2000?, 0x140000164f8?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x14000046510 sp=0x140000464e0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140000b2000, {0x140000164f8, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x14000046590 sp=0x14000046510 pc=0x14ccda678
net.(*netFD).Read(0x140000b2000, {0x140000164f8?, 0x14000074378?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x140000465e0 sp=0x14000046590 pc=0x14cd05048
net.(*conn).Read(0x14000010588, {0x140000164f8?, 0x14000166018?, 0x1400004668e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x14000046630 sp=0x140000465e0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x14000010588}, {0x140000164f8, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x14000046680 sp=0x14000046630 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x14000010588}, {0x140000164f8, 0x8, 0x8}, 0x14000046740)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x140000466f0 sp=0x14000046680 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x14000164000)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x140000467b0 sp=0x140000466f0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSession.func1()
        /workspace/srcdir/xrootd/session.go:104 +0x28 fp=0x140000467d0 sp=0x140000467b0 pc=0x14cd9d0c8
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000467d0 sp=0x140000467d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSession
        /workspace/srcdir/xrootd/session.go:104 +0x264

goroutine 39 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000047470 sp=0x14000047450 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x140000474b0 sp=0x14000047470 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4d28, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x140000474e0 sp=0x140000474b0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x14000168000?, 0x14000016550?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x14000047510 sp=0x140000474e0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000168000, {0x14000016550, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x14000047590 sp=0x14000047510 pc=0x14ccda678
net.(*netFD).Read(0x14000168000, {0x14000016550?, 0x14000074378?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x140000475e0 sp=0x14000047590 pc=0x14cd05048
net.(*conn).Read(0x1400019c018, {0x14000016550?, 0x14000074360?, 0x1400004768e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x14000047630 sp=0x140000475e0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x1400019c018}, {0x14000016550, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x14000047680 sp=0x14000047630 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x1400019c018}, {0x14000016550, 0x8, 0x8}, 0x14000047740)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x140000476f0 sp=0x14000047680 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x140001a20f0)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x140000477b0 sp=0x140000476f0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSubSession.func1()
        /workspace/srcdir/xrootd/session.go:471 +0x28 fp=0x140000477d0 sp=0x140000477b0 pc=0x14cd9f588
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000477d0 sp=0x140000477d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSubSession
        /workspace/srcdir/xrootd/session.go:471 +0x23c

goroutine 3239 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000255470 sp=0x14000255450 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x140002554b0 sp=0x14000255470 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4a58, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x140002554e0 sp=0x140002554b0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x14000192080?, 0x140000f8058?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x14000255510 sp=0x140002554e0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000192080, {0x140000f8058, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x14000255590 sp=0x14000255510 pc=0x14ccda678
net.(*netFD).Read(0x14000192080, {0x140000f8058?, 0x14000074378?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x140002555e0 sp=0x14000255590 pc=0x14cd05048
net.(*conn).Read(0x1400019c028, {0x140000f8058?, 0x14000074360?, 0x1400025568e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x14000255630 sp=0x140002555e0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x1400019c028}, {0x140000f8058, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x14000255680 sp=0x14000255630 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x1400019c028}, {0x140000f8058, 0x8, 0x8}, 0x14000255740)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x140002556f0 sp=0x14000255680 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x140001640f0)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x140002557b0 sp=0x140002556f0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSubSession.func1()
        /workspace/srcdir/xrootd/session.go:471 +0x28 fp=0x140002557d0 sp=0x140002557b0 pc=0x14cd9f588
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140002557d0 sp=0x140002557d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSubSession
        /workspace/srcdir/xrootd/session.go:471 +0x23c

goroutine 244 [GC worker (idle)]:
runtime.gopark(0x3c7fd72955efb?, 0x3?, 0x23?, 0x9?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000489f40 sp=0x14000489f20 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x14000489fd0 sp=0x14000489f40 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000489fd0 sp=0x14000489fd0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 243 [GC worker (idle)]:
runtime.gopark(0x3c7fd6cc9db32?, 0x3?, 0x6d?, 0xcc?, 0x140004b0798?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004b0740 sp=0x140004b0720 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004b07d0 sp=0x140004b0740 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b07d0 sp=0x140004b07d0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 162 [GC worker (idle)]:
runtime.gopark(0x3c7fded1ec3ec?, 0x3?, 0x8e?, 0xd?, 0x140004b0f98?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004b0f40 sp=0x140004b0f20 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004b0fd0 sp=0x140004b0f40 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b0fd0 sp=0x140004b0fd0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 239 [GC worker (idle)]:
runtime.gopark(0x3c7fd72955055?, 0x3?, 0x30?, 0x68?, 0x140004b1798?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004b1740 sp=0x140004b1720 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004b17d0 sp=0x140004b1740 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b17d0 sp=0x140004b17d0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 240 [GC worker (idle)]:
runtime.systemstack_switch()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:200 +0x8 fp=0x140004b1f40 sp=0x140004b1f30 pc=0x14cc98908
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1295 +0x224 fp=0x140004b1fd0 sp=0x140004b1f40 pc=0x14cc52ae4
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b1fd0 sp=0x140004b1fd0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 241 [GC worker (idle)]:
runtime.gopark(0x3c7fded1ec17b?, 0x3?, 0x71?, 0x67?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004b3740 sp=0x140004b3720 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004b37d0 sp=0x140004b3740 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b37d0 sp=0x140004b37d0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 242 [GC worker (idle)]:
runtime.gopark(0x14cf72de0?, 0x1?, 0x97?, 0x37?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004b3f40 sp=0x140004b3f20 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004b3fd0 sp=0x140004b3f40 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004b3fd0 sp=0x140004b3fd0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 259 [GC worker (idle)]:
runtime.gopark(0x3c7fd0279049e?, 0x1?, 0xfb?, 0xe6?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140004ac740 sp=0x140004ac720 pc=0x14cc6ec94
runtime.gcBgMarkWorker()
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140004ac7d0 sp=0x140004ac740 pc=0x14cc529ac
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140004ac7d0 sp=0x140004ac7d0 pc=0x14cc9af94
created by runtime.gcBgMarkStartWorkers
        /opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 302 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140000f5470 sp=0x140000f5450 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x140000f54b0 sp=0x140000f5470 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4c38, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x140000f54e0 sp=0x140000f54b0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x140000b2080?, 0x14000204000?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x140000f5510 sp=0x140000f54e0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140000b2080, {0x14000204000, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x140000f5590 sp=0x140000f5510 pc=0x14ccda678
net.(*netFD).Read(0x140000b2080, {0x14000204000?, 0x14000074378?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x140000f55e0 sp=0x140000f5590 pc=0x14cd05048
net.(*conn).Read(0x14000098018, {0x14000204000?, 0x14000074360?, 0x140000f568e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x140000f5630 sp=0x140000f55e0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x14000098018}, {0x14000204000, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x140000f5680 sp=0x140000f5630 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x14000098018}, {0x14000204000, 0x8, 0x8}, 0x140000f5740)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x140000f56f0 sp=0x140000f5680 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x140001a21e0)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x140000f57b0 sp=0x140000f56f0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSubSession.func1()
        /workspace/srcdir/xrootd/session.go:471 +0x28 fp=0x140000f57d0 sp=0x140000f57b0 pc=0x14cd9f588
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000f57d0 sp=0x140000f57d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSubSession
        /workspace/srcdir/xrootd/session.go:471 +0x23c

goroutine 2813 [IO wait]:
runtime.gopark(0xffffffffffffffff?, 0xffffffffffffffff?, 0x23?, 0x0?, 0x14cca5370?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140003d2470 sp=0x140003d2450 pc=0x14cc6ec94
runtime.netpollblock(0x0?, 0x0?, 0x0?)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:526 +0x160 fp=0x140003d24b0 sp=0x140003d2470 pc=0x14cc68380
internal/poll.runtime_pollWait(0x14cbd4b48, 0x72)
        /opt/x86_64-linux-musl/go/src/runtime/netpoll.go:305 +0xa0 fp=0x140003d24e0 sp=0x140003d24b0 pc=0x14cc95b80
internal/poll.(*pollDesc).wait(0x140000b2100?, 0x14000204030?, 0x0)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x140003d2510 sp=0x140003d24e0 pc=0x14ccd9d78
internal/poll.(*pollDesc).waitRead(...)
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140000b2100, {0x14000204030, 0x8, 0x8})
        /opt/x86_64-linux-musl/go/src/internal/poll/fd_unix.go:167 +0x1e8 fp=0x140003d2590 sp=0x140003d2510 pc=0x14ccda678
net.(*netFD).Read(0x140000b2100, {0x14000204030?, 0x14000074378?, 0x14cd99c78?})
        /opt/x86_64-linux-musl/go/src/net/fd_posix.go:55 +0x28 fp=0x140003d25e0 sp=0x140003d2590 pc=0x14cd05048
net.(*conn).Read(0x14000010008, {0x14000204030?, 0x14000074360?, 0x140003d268e?})
        /opt/x86_64-linux-musl/go/src/net/net.go:183 +0x34 fp=0x140003d2630 sp=0x140003d25e0 pc=0x14cd107f4
io.ReadAtLeast({0x14ce34b28, 0x14000010008}, {0x14000204030, 0x8, 0x8}, 0x8)
        /opt/x86_64-linux-musl/go/src/io/io.go:332 +0xa8 fp=0x140003d2680 sp=0x140003d2630 pc=0x14cca5798
io.ReadFull(...)
        /opt/x86_64-linux-musl/go/src/io/io.go:351
go-hep.org/x/hep/xrootd/xrdproto.ReadResponseWithReuse({0x14ce34b28, 0x14000010008}, {0x14000204030, 0x8, 0x8}, 0x140003d2740)
        /workspace/srcdir/xrootd/xrdproto/xrdproto.go:398 +0x38 fp=0x140003d26f0 sp=0x140003d2680 pc=0x14cd91298
go-hep.org/x/hep/xrootd.(*cliSession).consume(0x140001a22d0)
        /workspace/srcdir/xrootd/session.go:233 +0xa0 fp=0x140003d27b0 sp=0x140003d26f0 pc=0x14cd9dba0
go-hep.org/x/hep/xrootd.newSubSession.func1()
        /workspace/srcdir/xrootd/session.go:471 +0x28 fp=0x140003d27d0 sp=0x140003d27b0 pc=0x14cd9f588
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140003d27d0 sp=0x140003d27d0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.newSubSession
        /workspace/srcdir/xrootd/session.go:471 +0x23c

goroutine 4634 [select]:
runtime.gopark(0x140003d4fa8?, 0x2?, 0x0?, 0x0?, 0x140003d4f9c?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140003d4e40 sp=0x140003d4e20 pc=0x14cc6ec94
runtime.selectgo(0x140003d4fa8, 0x140003d4f98, 0x0?, 0x1, 0x0?, 0x1)
        /opt/x86_64-linux-musl/go/src/runtime/select.go:328 +0x6b4 fp=0x140003d4f50 sp=0x140003d4e40 pc=0x14cc7eba4
go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID.func1()
        /workspace/srcdir/xrootd/session.go:423 +0x7c fp=0x140003d4fd0 sp=0x140003d4f50 pc=0x14cd9ef6c
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140003d4fd0 sp=0x140003d4fd0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID
        /workspace/srcdir/xrootd/session.go:422 +0x74

goroutine 4483 [select]:
runtime.gopark(0x14000336fa8?, 0x2?, 0x0?, 0x0?, 0x14000336f9c?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000336e40 sp=0x14000336e20 pc=0x14cc6ec94
runtime.selectgo(0x14000336fa8, 0x14000336f98, 0x0?, 0x1, 0x0?, 0x1)
        /opt/x86_64-linux-musl/go/src/runtime/select.go:328 +0x6b4 fp=0x14000336f50 sp=0x14000336e40 pc=0x14cc7eba4
go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID.func1()
        /workspace/srcdir/xrootd/session.go:423 +0x7c fp=0x14000336fd0 sp=0x14000336f50 pc=0x14cd9ef6c
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000336fd0 sp=0x14000336fd0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID
        /workspace/srcdir/xrootd/session.go:422 +0x74

goroutine 4635 [select]:
runtime.gopark(0x140000aafa8?, 0x2?, 0x9?, 0x18?, 0x140000aaf9c?)
        /opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x140000aae40 sp=0x140000aae20 pc=0x14cc6ec94
runtime.selectgo(0x140000aafa8, 0x140000aaf98, 0x0?, 0x1, 0x0?, 0x1)
        /opt/x86_64-linux-musl/go/src/runtime/select.go:328 +0x6b4 fp=0x140000aaf50 sp=0x140000aae40 pc=0x14cc7eba4
go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID.func1()
        /workspace/srcdir/xrootd/session.go:423 +0x7c fp=0x140000aafd0 sp=0x140000aaf50 pc=0x14cd9ef6c
runtime.goexit()
        /opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000aafd0 sp=0x140000aafd0 pc=0x14cc9af94
created by go-hep.org/x/hep/xrootd.(*cliSession).unclaimPathID
        /workspace/srcdir/xrootd/session.go:422 +0x74

[66921] signal (6): Abort trap: 6
in expression starting at /Users/mato/Development/EDM4hep.jl/examples/FCC/analysis_MT.jl:126
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 19223193 (Pool: 19187879; Big: 35314); GC: 116
Moelf commented 6 months ago

sigh... do you know when will CERN / tier-2s support Sci-Token and HTTP based methods? We want to avoid xrootd for multiple reasons, another reason is Go-xrootd doesn't support GSI authenticaction

peremato commented 6 months ago

I have a reproducer for this crash (out-of-band (OOB) data signal ??) for macOS. It is very strange.

using UnROOT

f = "root://eospublic.cern.ch//eos/experiment/fcc/ee/generation/DelphesEvents/winter2023/IDEA/p8_ee_ZZ_ecm240/events_000189367.root"
tfile = ROOTFile(f)
tree = tfile["events"]

# select branches that are vector<>
vpattern = r"(std::)?vector<(std::)?(.*)>"
branches = []
for (i,key) in enumerate(keys(tree))
    classname = tree.fBranches[i].fClassName
    isnothing(match(vpattern, classname)) && continue
    push!(branches, key)
end
events = LazyTree(tfile, "events", branches)

const N = Threads.nthreads()
total_E = Threads.Atomic{Float32}(0)
nsel = Threads.Atomic{Int64}(0)

function do_analysis_threads(events)
    Threads.@threads for evt in events
        μindices = evt.var"Muon#0_index"
        length(μindices) != 2 && continue
        energies = evt.ReconstructedParticles_energy
        Threads.atomic_add!(nsel, 1)
        Threads.atomic_add!(total_E, sum(getindex.(Ref(energies), μindices .+ 1)))
    end
end

@info "Threads $N"
@time do_analysis_threads(events)
println("<Eμ>/event $(total_E.value/nsel.value)GeV")

If you comment the line @info or do not use the interpolation $N the reproducer runs successfully.

Moelf commented 6 months ago

I copied that snippet unmodified, and it doesn't seem to crash:

akako@frame ~/.j/d/UnROOT (remove_threadcall)> julia --project ./issue311.jl
[ Info: Threads 4
  6.794072 seconds (902.22 k allocations: 177.747 MiB, 0.21% gc time, 26.29% compilation time)
<Eμ>/event 95.97867GeV
peremato commented 6 months ago

Where did you run it? In lxplus (alma9) it didn't fail for me. Perhaps is very OS dependent.

Moelf commented 6 months ago

I'm running on Arch Linux with the binary installed from juliaup.

I just tried again on 

[09:47] login04.af.uchicago.edu:~/.julia/dev/UnROOT $ cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"

[09:48] login04.af.uchicago.edu:~/.julia/dev/UnROOT $ julia --project issue311.jl
The latest version of Julia in the `release` channel is 1.10.2+0.x64.linux.gnu. You currently have `1.10.1+0.x64.linux.gnu` installed. Run:

  juliaup update

in your terminal shell to install Julia 1.10.2+0.x64.linux.gnu and update the `release` channel to that version.
[ Info: Threads 8
  7.912776 seconds (902.70 k allocations: 190.557 MiB, 0.69% gc time, 54.08% compilation time)
<Eμ>/event 95.97873GeV

still works just fine.

Maybe you should try to use a Julia version not distributed via LCG, I have noticed issues with the LD_* overload causing weird stuff from time to time