search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.84k stars 901 forks source link

gossipd crash in v24.02 #7309

Closed alecalve closed 5 months ago

alecalve commented 5 months ago

Issue and Steps to Reproduce

Gist with full logs: https://gist.github.com/alecalve/ee7abc58aeeb24024b3c73f7ea452fa6

lightning_gossipd: gossip_store: bad checksum offset 3044: ... (see gist) ...
0x555fb8de25d4 send_backtrace
    common/daemon.c:33
0x555fb8debbfc status_failed
    common/status.c:221
0x555fb8dd8b99 gossip_store_get_with_hdr
    gossipd/gossip_store.c:480
0x555fb8dd8bc7 check_msg_type
    gossipd/gossip_store.c:491
0x555fb8dd8d4a gossip_store_set_flag
    gossipd/gossip_store.c:509
0x555fb8dd8f51 gossip_store_del
    gossipd/gossip_store.c:561
0x555fb8dd9756 process_node_announcement
    gossipd/gossmap_manage.c:917
0x555fb8ddb33e gossmap_manage_node_announcement
    gossipd/gossmap_manage.c:1007
0x555fb8dd78f5 handle_recv_gossip
    gossipd/gossipd.c:222
0x555fb8dd79ef connectd_req
    gossipd/gossipd.c:313
0x555fb8de28c2 handle_read
    common/daemon_conn.c:35
0x555fb8f21086 next_plan
    ccan/ccan/io/io.c:59
0x555fb8f2150d do_plan
    ccan/ccan/io/io.c:407
0x555fb8f215a6 io_ready
    ccan/ccan/io/io.c:417
0x555fb8f22e06 io_loop
    ccan/ccan/io/poll.c:453
0x555fb8dd7cb6 main
    gossipd/gossipd.c:684
0x7ff535f7c6d0 ???
    ???:0
2024-05-14T05:43:21.046Z **BROKEN** gossipd: gossip_store: bad checksum offset 3044: ... (see gist) ...
2024-05-14T05:43:21.049Z **BROKEN** gossipd: backtrace: common/daemon.c:38 (send_backtrace) 0x555fb8de261c
2024-05-14T05:43:21.049Z **BROKEN** gossipd: backtrace: common/status.c:221 (status_failed) 0x555fb8debbfc
2024-05-14T05:43:21.049Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:480 (gossip_store_get_with_hdr) 0x555fb8dd8b99
2024-05-14T05:43:21.049Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:491 (check_msg_type) 0x555fb8dd8bc7
2024-05-14T05:43:21.050Z **BROKEN** connectd: STATUS_FAIL_GOSSIP_IO: gossipd exited?
lightningd: connectd failed (exit status 242), exiting.

I restarted the node and it's working so far.

vincenzopalazzo commented 5 months ago

Can you check if this happens with v24.02.2

alecalve commented 5 months ago

I can update to that version, but I can't replicate the crash.

vincenzopalazzo commented 5 months ago

I can update to that version, but I can't replicate the crash.

Nice, so it should be fixed in the recent version :)

Feel free to reopen is the crash happen again

kallerosenbaum commented 5 months ago

This seems to have happened to me too. I'm running v24.02.1, and it won't restart. Maybe this issue should be reopened @vincenzopalazzo?

2024-05-26T07:59:09.828Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 694409x784x0
2024-05-26T07:59:09.950Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 696269x1725x1
2024-05-26T07:59:09.950Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 697349x748x1
2024-05-26T07:59:10.078Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 704814x1283x0
2024-05-26T07:59:10.078Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 705648x2355x1
2024-05-26T07:59:10.078Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 708849x911x0
2024-05-26T07:59:12.766Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 556816x1909x0
2024-05-26T07:59:12.930Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 556817x932x1
2024-05-26T07:59:12.930Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 558388x1246x0
2024-05-26T07:59:12.930Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 711129x1434x1
2024-05-26T07:59:12.930Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 681773x1105x1
2024-05-26T07:59:12.930Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 681777x1124x1
2024-05-26T08:07:47.444Z INFO    030f83907a86bd674f323e0d2c735f714a0b1ae2c10480a392555e8eb6c1777b87-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 708163x1056x0
2024-05-26T08:50:09.804Z UNUSUAL 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b-lightningd: Unknown channel 62c9218fc04b89cc5771726d679b5d3bbee267e9307a30cd4cb4625a329bcd50 for WIRE_CHANNEL_REESTABLISH
2024-05-26T09:14:49.570Z INFO    03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f-channeld-chan#68770: Peer connection lost
2024-05-26T09:14:49.570Z INFO    03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f-chan#68770: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2024-05-26T09:53:10.323Z UNUSUAL 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b-lightningd: Unknown channel 62c9218fc04b89cc5771726d679b5d3bbee267e9307a30cd4cb4625a329bcd50 for WIRE_CHANNEL_REESTABLISH
2024-05-26T10:17:57.596Z **BROKEN** gossipd: gossip_store: bad checksum offset 671: 58395ef738fe63cd414fc7e6defa9d6fdb707eee9b8665c7b705f50cb91cb48c12d43673d202758fdf228604276c5e76c3ebe98c237400006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000b43c900091100000225e29bcf897f
22d9f546ad71185f9519db538d484b2fd2ea2be02e20b5c7224b03afa7a8196dbca763ee6f9a34b634a7adc03f154e5d6979fe654db5606b5fb2b10351ca4462ba0f384e2691d871a3cca46bedf47db07fd9d10e12bc77a25539e65503f801b9fad3c604215665d162e343b4e66d771eaefcf8333014a2ca01ce550b208000000a911183f600000000100500000000000f
4240000001b0c5dce21564af01b8010006bb437a683e2a5af7630469482aa4e767726e1ae87bcd01df29ff25e9479cf0063f5de535cb33380036271e487e098b7e6417972e52baaefd798a759d43a67f1db44c6e980be5d17620aabe144128ec0af197745c1e15563919703ab5922cc4365cc8228cfda3c57d0fa07afcf4ce39025b6fb61e04e5fe79b12fc10d799bfa51
ee77015c8ada10921dfddb2dc8bbd514400d303c722194c547415e272112ea264d37434a6e727c9f857c3c1e0e52d7be41f72ca0ace2c57ea868484cfed7aa6e014f415d9e324658c4fdd7f3caccee50610cce2ebe5e19a730948cb4101ac96c5691da14b91642939bfd80ec318dc9ec92bc8054bcb3d11abc352420c6fb4000006fe28c0ab6f1b372c1a6a246ae63f74f
931e8365e15a089c68d61900000000000bf7bd0001a200000225e29bcf897f22d9f546ad71185f9519db538d484b2fd2ea2be02e20b5c7224b0290788
...
e15a089c68d61900000000000c21a90005a7000002ce7abb5e625990f7898520bca721b7ae316165eef81133dcfe4068fcfbe37ac6037a78b7b7c3e618b42c5377b516b26f5e4654b828edc13809cd7cfb5872131d4703e8a8be5f0e7562838bc7b6baeef477eac3cefa08e5509a2ce78bb32045ab751a03d2bef4d2f40f7c8178dd15588f6a7d200ffde2252ef5b0e986
43aad83ebb90728000000a22672c52000000001005000000000052f5b5000001b08962c79b64b7a44c01002122ea8daa722cb26239cf628a9110dc1be4636ced6349d1f477ef39c60127885b4cb9ea419c06009d635c0a6cf14d02ec0b26eff3185ca1eab0eddf2be33a52303a53ab02cf3fa972544d716417ba00d9bc969ba3506dbf90da4bffaec71db70d88f90f932e
b8717466a01d187a541b8112d3d9eb5c9ca89853cbe0537453d5855b2bb08e4f2bb179740326d49eacfc3b1363e521acf671dc9e4b3846fcfa147177ffdba991c8f37458bb40f0ea2123107fe30853473df6c7272772713814aa57285173258d6c94398814a3784a3a531b9630e06c1df9a037f844b37979811e7fffacf431374354a235ae7e60955c480729331456a6d6
a2e4c9dddf5fa15c3900006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000c1a5200020300000212dfb5138fce06681efa58db32cfdb4f940b161f7385de55c236126752a7aaad0255a746e648181de938d66a31bfe873c7ce2b652a8978189770e19c4337bede530395b396b98e30157ea9c30de1aea80f11c8c0e3cf9e3e22bdb720e0
26102afedb0232df667d45c78285631daba9b5c068e65259c3447e251f0daa0031bb62dec3d20000000a911183f600000000100500000000000f4240000001b0379c392864b7a44c010085a28fba759a81 (version v24.02.1)
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: common/daemon.c:38 (send_backtrace) 0x560a0b38d1fc
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: common/status.c:221 (status_failed) 0x560a0b396ce1
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:480 (gossip_store_get_with_hdr) 0x560a0b3832d2
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:491 (check_msg_type) 0x560a0b383300
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:509 (gossip_store_set_flag) 0x560a0b383493
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:561 (gossip_store_del) 0x560a0b3836df
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:767 (process_channel_update) 0x560a0b384f06
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:876 (gossmap_manage_channel_update) 0x560a0b3856de
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:217 (handle_recv_gossip) 0x560a0b381fef
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:313 (connectd_req) 0x560a0b382124
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: common/daemon_conn.c:35 (handle_read) 0x560a0b38d4ca
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x560a0b4d4823
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x560a0b4d4cfe
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x560a0b4d4d9b
2024-05-26T10:17:57.660Z **BROKEN** gossipd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x560a0b4d6738
2024-05-26T10:17:57.661Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:684 (main) 0x560a0b3823e8
2024-05-26T10:17:57.661Z **BROKEN** gossipd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x7fe1390bed8f
2024-05-26T10:17:57.661Z **BROKEN** gossipd: backtrace: ../csu/libc-start.c:392 (__libc_start_main_impl) 0x7fe1390bee3f
2024-05-26T10:17:57.661Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x560a0b37e7c4
2024-05-26T10:17:57.661Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2024-05-26T10:17:57.661Z **BROKEN** gossipd: STATUS_FAIL_INTERNAL_ERROR: gossip_store: bad checksum offset 671: 58395ef738fe63cd414fc7e6defa9d6fdb707eee9b8665c7b705f50cb91cb48c12d43673d202758fdf228604276c5e76c3ebe98c237400006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000b
43c900091100000225e29bcf897f22d9f546ad71185f9519db538d484b2fd2ea2be02e20b5c7224b03afa7a8196dbca763ee6f9a34b634a7adc03f154e5d6979fe654db5606b5fb2b10351ca4462ba0f384e2691d871a3cca46bedf47db07fd9d10e12bc77a25539e65503f801b9fad3c604215665d162e343b4e66d771eaefcf8333014a2ca01ce550b208000000a9111
83f600000000100500000000000f4240000001b0c5dce21564af01b8010006bb437a683e2a5af7630469482aa4e767726e1ae87bcd01df29ff25e9479cf0063f5de535cb33380036271e487e098b7e6417972e52baaefd798a759d43a67f1db44c6e980be5d17620aabe144128ec0af197745c1e15563919703ab5922cc4365cc8228cfda3c57d0fa07afcf4ce39025b6f
...
588f6a7d200ffde2252ef5b0e98643aad83ebb90728000000a22672c52000000001005000000000052f5b5000001b08962c79b64b7a44c01002122ea8daa722cb26239cf628a9110dc1be4636ced6349d1f477ef39c60127885b4cb9ea419c06009d635c0a6cf14d02ec0b26eff3185ca1eab0eddf2be33a52303a53ab02cf3fa972544d716417ba00d9bc969ba3506dbf
90da4bffaec71db70d88f90f932eb8717466a01d187a541b8112d3d9eb5c9ca89853cbe0537453d5855b2bb08e4f2bb179740326d49eacfc3b1363e521acf671dc9e4b3846fcfa147177ffdba991c8f37458bb40f0ea2123107fe30853473df6c7272772713814aa57285173258d6c94398814a3784a3a531b9630e06c1df9a037f844b37979811e7fffacf431374354a2
35ae7e60955c480729331456a6d6a2e4c9dddf5fa15c3900006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000c1a5200020300000212dfb5138fce06681efa58db32cfdb4f940b161f7385de55c236126752a7aaad0255a746e648181de938d66a31bfe873c7ce2b652a8978189770e19c4337bede530395b396b98e30157ea9c30de1ae
a80f11c8c0e3cf9e3e22bdb720e026102afedb0232df667d45c78285631daba9b5c068e65259c3447e251f0daa0031bb62dec3d20000000a911183f600000000100500000000000f4240000001b0379c392864b7a44c010085a28fba759a81

2024-05-26T10:18:29.752Z INFO    lightningd: v24.02.1

2024-05-26T10:19:02.280Z INFO    lightningd: v24.02.1

2024-05-26T10:19:32.958Z INFO    lightningd: v24.02.1
...
kallerosenbaum commented 5 months ago

I upgraded to 24.02.2 since @vincenzopalazzo suggested that above and now it runs (after I deleted a clboss config that's unknown because I don't load the clboss plugin anymore. Apparently the policy on unknown options have changed?).

I guess we don't need to reopen this issue.

vincenzopalazzo commented 5 months ago

@kallerosenbaum probably but feel free to ping me if you have any similar cases with the 24.02.2 😄