v23.02 crashes at start on freebsd

[denis2342]

I did try the new v23.02 yesterday. It crashed at start but I thought it has to do with my tunnels to the unix socket rpc. tried again and it did run flawlessly until I restarted it today (no tunnels used!).

But it crashes at the start. I disabled all plugins, still the same.

OS: freebsd 13.2 hardware: dual intel xeon, 64gb

this is the output:

 sudo -u c-lightning lightningd --conf /usr/local/etc/lightningd-bitcoin.conf
Password:
lightning_connectd: FATAL SIGNAL 6 (version v23.02)
0x34ea5a send_backtrace
    common/daemon.c:33
0x34ec89 crashdump
    common/daemon.c:46
0x82535fb5f handle_signal
    /usr/src/lib/libthr/thread/thr_sig.c:303
0x82535f11e thr_sighandler
    /usr/src/lib/libthr/thread/thr_sig.c:246
0x7ffffffff8a2 ???
    ???:0
0x824ac2c5a ???
    /usr/obj/usr/src/amd64.amd64/lib/libc/thr_kill.S:4
0x824a3b6d3 __raise
    /usr/src/lib/libc/gen/raise.c:52
0x824aeca58 abort
    /usr/src/lib/libc/stdlib/abort.c:67
0x3e1700 call_error
    ccan/ccan/tal/tal.c:93
0x3e1700 check_bounds
    ccan/ccan/tal/tal.c:165
0x3e1700 to_tal_hdr
    ccan/ccan/tal/tal.c:174
0x3e1211 to_tal_hdr_or_null
    ccan/ccan/tal/tal.c:186
0x3e1211 tal_alloc_
    ccan/ccan/tal/tal.c:426
0x3db8f4 io_new_conn_
    ccan/ccan/io/io.c:91
0x3dd2e1 accept_conn
    ccan/ccan/io/poll.c:277
0x3dd2e1 io_loop
    ccan/ccan/io/poll.c:444
0x3419fa main
    connectd/connectd.c:2081
lightning_connectd: FATAL SIGNAL (version v23.02)
0x34ea5a send_backtrace
    common/daemon.c:33
0x35811b status_failed
    common/status.c:221
0x358384 status_backtrace_exit
    common/subdaemon.c:18
0x34ec8f crashdump
    common/daemon.c:49
0x82535fb5f handle_signal
    /usr/src/lib/libthr/thread/thr_sig.c:303
0x82535f11e thr_sighandler
    /usr/src/lib/libthr/thread/thr_sig.c:246
0x7ffffffff8a2 ???
    ???:0
0x824ac2c5a ???
    /usr/obj/usr/src/amd64.amd64/lib/libc/thr_kill.S:4
0x824a3b6d3 __raise
    /usr/src/lib/libc/gen/raise.c:52
0x824aeca58 abort
    /usr/src/lib/libc/stdlib/abort.c:67
0x3e1700 call_error
    ccan/ccan/tal/tal.c:93
0x3e1700 check_bounds
    ccan/ccan/tal/tal.c:165
0x3e1700 to_tal_hdr
    ccan/ccan/tal/tal.c:174
0x3e1211 to_tal_hdr_or_null
    ccan/ccan/tal/tal.c:186
0x3e1211 tal_alloc_
    ccan/ccan/tal/tal.c:426
0x3db8f4 io_new_conn_
    ccan/ccan/io/io.c:91
0x3dd2e1 accept_conn
    ccan/ccan/io/poll.c:277
0x3dd2e1 io_loop
    ccan/ccan/io/poll.c:444
0x3419fa main
    connectd/connectd.c:2081
lightningd: gossipd failed (exit status 2), exiting.
Lost connection to the RPC socket.

[denis2342]

I can start it in --offline mode and that works. can even do getinfo, listfunds and listpeers.

[denis2342]

oldest commit I tried now was 38d90b250596d38a777ab064c3788a730f14d753, which has the same problem. Can't bisect because of DB version.

also tried commit beec517 with cherry-pick e315f30 for the db upgrade, still the same error. can't go back more.

[denis2342]

after dozens of tries it started again. stopped it after a while to turn off debug logging and then it needed another 20-30 tries to start. looks like some race condition in the beginning

[denis2342]

yeah, #6070 does the trick.