search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.82k stars 896 forks source link

gossipd crash: gossip_store: get delete entry offset #7142

Open vincenzopalazzo opened 6 months ago

vincenzopalazzo commented 6 months ago 
2024-03-10T10:58:21.974Z DEBUG   03d35779ff612c574b92494d74300ce467e0eb510181a6ae3eb74d8ada891d82aa-gossipd: Received node_announcement for node 03dbd8ef074d87608ea6ceb905e54cdc5bff505b649b34c695850f27beb815bcaa
2024-03-10T10:58:21.974Z **BROKEN** gossipd: gossip_store: get delete entry offset 387938967/388133826 (version v24.02-modded)
2024-03-10T10:58:21.974Z **BROKEN** gossipd: backtrace: common/daemon.c:38 (send_backtrace) 0x561dffb59322
2024-03-10T10:58:21.974Z **BROKEN** gossipd: backtrace: common/status.c:221 (status_failed) 0x561dffb62a87
2024-03-10T10:58:21.974Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:466 (gossip_store_get_with_hdr) 0x561dffb4f6ee
2024-03-10T10:58:21.974Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:491 (check_msg_type) 0x561dffb4f768
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:509 (gossip_store_set_flag) 0x561dffb4f8eb
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:561 (gossip_store_del) 0x561dffb4fb31
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:917 (process_node_announcement) 0x561dffb50359
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:1115 (reprocess_queued_msgs) 0x561dffb516e6
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:673 (gossmap_manage_handle_get_txout_reply) 0x561dffb5187f
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:586 (recv_req) 0x561dffb4de59
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: common/daemon_conn.c:35 (handle_read) 0x561dffb595cd
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x561dffbec306
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x561dffbec78d
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x561dffbec826
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x561dffbee115
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:684 (main) 0x561dffb4e857
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7f3224bb1d09
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x561dffb4ad39
2024-03-10T10:58:21.975Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2024-03-10T10:58:21.975Z **BROKEN** gossipd: STATUS_FAIL_INTERNAL_ERROR: gossip_store: get delete entry offset 387938967/388133826

I am on the v24.02 and idk if this was already fixed, but just posting here to keep track of the stacktrace

Dominion5254 commented 2 weeks ago

I am seeing the similar messages followed by lightningd: gossipd failed (exit status 242), exiting. on v24.08. Happy to share more logs if needed.

Deleting the gossip_store and restarting CLN seems to resolve the issue (until/if it occurs again), but it would be preferred if CLN could prevent or at least detect and deal with the issue (corruption?) rather than crashing.