Failure to parse a pending_node_announcement when loading the gossip_store

[thestick613]

I've been having trouble with this node. I have two nodes, they had a channel between them. I was able to invoice from A and pay from B, but every invoice generated from B failed to get payed from A. They were small invoices, like 2000 sat each. I investigated the channel status using the output i learned from #2709, and the healthy node changed the values of msatoshi_to_us, their_channel_reserve_satoshis and our_channel_reserve_satoshis after every payment, but the 'sick' one didn't. I decided to close all the channels of the bad node, because i was running on mainnet funds. After i closed the last channel, the node crashed:

lightning_gossipd: pending node_announcement 01014db000711e286148c285a8565ecafcf24a8955ad12d96d338f9cf67fb708fa3225fee739a37f1eddecaf041b79e60f1f7790630672e921be58e3eaf71749fc4900005cc1a7d9028b6dc171f6034
67690f799b6a0d52b1f5a5b8977b1453440016ea7917f53925f028b6d686f73743463727970746f2e636f6d000000000000000000000000000000000000070150f1d8cc2607 malformed? (version v0.7.0-490-gbbedd38)
0x410fc4 send_backtrace
        common/daemon.c:40
0x413a54 status_failed
        common/status.c:192
0x40970f process_pending_node_announcement
        gossipd/routing.c:1315
0x409b32 routing_add_channel_update
        gossipd/routing.c:1891
0x406652 gossip_store_load
        gossipd/gossip_store.c:582
0x40e36d gossip_init
        gossipd/gossipd.c:1862
0x40f2da recv_req
        gossipd/gossipd.c:2702
0x41111a handle_read
        common/daemon_conn.c:31
0x427521 next_plan
        ccan/ccan/io/io.c:59
0x4279e1 do_plan
        ccan/ccan/io/io.c:407
0x427a0c io_ready
        ccan/ccan/io/io.c:417
0x4290b5 io_loop
        ccan/ccan/io/poll.c:445
0x40f515 main
        gossipd/gossipd.c:2827
0x7faae2d2b82f ???
        ???:0
0x402b18 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2019-06-06T07:46:23.887Z lightning_gossipd(15217): gossip_store node_announcement 2480389 replaces 2480389!
2019-06-06T07:46:23.889Z lightning_gossipd(15217): pending node_announcement 01014db000711e286148c285a8565ecafcf24a8955ad12d96d338f9cf67fb708fa3225fee739a37f1eddecaf041b79e60f1f7790630672e921be58e3eaf71749fc4900005cc1a7d9028b6dc171f603467690f799b6a0d52b1f5a5b8977b1453440016ea7917f53925f028b6d686f73743463727970746f2e636f6d000000000000000000000000000000000000070150f1d8cc2607 malformed? (version v0.7.0-490-gbbedd38)
2019-06-06T07:46:23.889Z lightning_gossipd(15217): backtrace: common/daemon.c:45 (send_backtrace) 0x411008
2019-06-06T07:46:23.889Z lightning_gossipd(15217): backtrace: common/status.c:192 (status_failed) 0x413a54
2019-06-06T07:46:23.889Z lightning_gossipd(15217): backtrace: gossipd/routing.c:1315 (process_pending_node_announcement) 0x40970f
2019-06-06T07:46:23.889Z lightning_gossipd(15217): backtrace: gossipd/routing.c:1891 (routing_add_channel_update) 0x409b32
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: gossipd/gossip_store.c:582 (gossip_store_load) 0x406652
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: gossipd/gossipd.c:1862 (gossip_init) 0x40e36d
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: gossipd/gossipd.c:2702 (recv_req) 0x40f2da
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: common/daemon_conn.c:31 (handle_read) 0x41111a
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x427521
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x4279e1
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x427a0c
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x4290b5
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: gossipd/gossipd.c:2827 (main) 0x40f515
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: (null):0 ((null)) 0x7faae2d2b82f
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: (null):0 ((null)) 0x402b18
2019-06-06T07:46:23.890Z lightning_gossipd(15217): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-06-06T07:46:23.890Z lightning_gossipd(15217): STATUS_FAIL_INTERNAL_ERROR: pending node_announcement 01014db000711e286148c285a8565ecafcf24a8955ad12d96d338f9cf67fb708fa3225fee739a37f1eddecaf041b79e60f1f7790630672e921be58e3eaf71749fc4900005cc1a7d9028b6dc171f603467690f799b6a0d52b1f5a5b8977b1453440016ea7917f53925f028b6d686f73743463727970746f2e636f6d000000000000000000000000000000000000070150f1d8cc2607 malformed?
lightningd: lightning_gossipd failed (exit status 242), exiting.

There are some small funds in there, that i would like to recover.

[SimonVrouwe]

Thanks for reporting this. Your crash is probably unrelated to any payments and is purely a gossip processing error. Looks you have a recent version bbedd38, there where recent optimizations changes to gossip so perhaps its related to that.

Would it be possible for you to upload the original gossip_store file (as right after crash)? Then we can try to reproduce the error. The file is located in ~/.lightning directory. To upload the file, just create new comment and see bottom bar "Attach files ...", you need to compress zip or tar.gz

You can restart the node, but it could crash again when receiving troubling gossip sequence.

[thestick613]

Is it safe to publish it online?

[SimonVrouwe]

Good question, the file includes your private (or local) channel updates. So that would be relevant if you have private channels. For the rest it contains public gossip messages, like WIRE_CHANNEL_ANNOUNCEMENT, WIRE_CHANNEL_UPDATE etc. as all cases inside this switch: https://github.com/ElementsProject/lightning/blob/bbedd3819d0eb0047c0c3cb4e8f74b9ab6763e4a/gossipd/gossip_store.c#L546-L547

edit: and indeed local channel updates in the gossip_store could link your node to you

[cdecker]

I am encountering the same issue (gossip_store attached). And I traced it down to this condition:

https://github.com/ElementsProject/lightning/blob/1ca0eef744f4f01148ce804ac4e7e5b826b0176e/gossipd/routing.c#L2145-L2148

With this backtrace:

#0  routing_add_node_announcement (rstate=0x55d6d18f5228, msg=0x55d6d1dc05b8 "\001\001d\023\253g\333\337\177\034\002.2n\312\024H\262ߪ\020Fs\b\020\312k\030\207\304\370\265@\220z\223\217w\373\252a1\336\001H+u\237\061\317\377\300\313", <incomplete sequence \366>, index=9128969)
    at gossipd/routing.c:2144
#1  0x000055d6cfda2e39 in process_pending_node_announcement (rstate=0x55d6d18f5228, nodeid=0x55d6d1dde818) at gossipd/routing.c:1312
#2  0x000055d6cfda43e9 in routing_add_channel_update (rstate=0x55d6d18f5228,
    update=0x55d6d18f5828 "\001\002֗\316m5v\262\224\267\272%\034Z|\324ۓn\276\332\067\235\266g\021U\032\327\062͐\252\032\254!\211I=\371\223~=[\212\242lh\020\247\274C\373x>\016uz\203u\256\267d\242?o\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031", in
dex=11205043) at gossipd/routing.c:1892
#3  0x000055d6cfd9f124 in gossip_store_load (rstate=0x55d6d18f5228, gs=0x55d6d18f5528) at gossipd/gossip_store.c:582
#4  0x000055d6cfdaa72d in gossip_init (conn=0x55d6d18f48b8, daemon=0x55d6d18f4438, msg=0x55d6d18f5128 "\v\270o\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031") at gossipd/gossipd.c:1862
#5  0x000055d6cfdac9da in recv_req (conn=0x55d6d18f48b8, msg=0x55d6d18f5128 "\v\270o\342\214\n\266\361\263r\301\246\242F\256c\367O\223\036\203e\341Z\b\234h\326\031", daemon=0x55d6d18f4438) at gossipd/gossipd.c:2702
#6  0x000055d6cfdafdf7 in handle_read (conn=0x55d6d18f48b8, dc=0x55d6d18f4668) at common/daemon_conn.c:31
#7  0x000055d6cfdd34cc in next_plan (conn=0x55d6d18f48b8, plan=0x55d6d18f48d8) at ccan/ccan/io/io.c:59
#8  0x000055d6cfdd4049 in do_plan (conn=0x55d6d18f48b8, plan=0x55d6d18f48d8, idle_on_epipe=false) at ccan/ccan/io/io.c:407
#9  0x000055d6cfdd4087 in io_ready (conn=0x55d6d18f48b8, pollflags=1) at ccan/ccan/io/io.c:417
#10 0x000055d6cfdd60dc in io_loop (timers=0x55d6d18f44a8, expired=0x7ffea6682300) at ccan/ccan/io/poll.c:445
#11 0x000055d6cfdacd20 in main (argc=2, argv=0x7ffea6682408) at gossipd/gossipd.c:2827

[rustyrussell]

I am encountering the same issue (gossip_store attached). And I traced it down to this condition:

Can't see attachment?

Since you both upgraded partway through the gossip_store changed, and then again to master, I suspect that's the cause, since we had some broken gossip stuff in there.

Workaround is to delete ~/.lightning/gossip_store and restart. Fix is for us to bump the version so this happens automatically when people upgrade, in case they've been tracking master.

[SimonVrouwe]

I restartes my mainnet node (e902d9a) a few days ago with a clean gossip_store and it was running without issues for a few days and dev-compacting-gossip-store worked fine. Snipped from 1st logfile:

2019-06-04T06:37:52.088Z lightning_gossipd(3817): total store load time: 0 msec
2019-06-04T06:37:52.088Z lightning_gossipd(3817): gossip_store: Read 0/0/0/0 cannounce/cupdate/nannounce/cdelete from store (0 deleted) in 1 bytes
...
2019-06-04T06:37:55.130Z lightningd(3797): Server started with public key 02682020a37a3b5876b16bffbf0ac56d7f379695a5d273cf16ed0f35f18ab2dc98, alias LookBusy4 (color #026820) and lightningd v0.7.0-471-ge902d9a

Trying to reproduce this issue, I stopped the node, copied gossip_store to gossip_store_afterstop_jun07.zip and started the node (with new 2nd logfile) and it crashed showing in the terminal:

lightning_gossipd: pending node_announcement 0101cd42b588499d54b9a95c48927f52951dd465dd9244f0cbb91a4035763a185a9e20c5b33bdf30ac67be2867faf4847367f1a4a65534c83d23fa124933b8993c1700005cf16af203cbea26f993aa7873c16798a161bc6cd418180c81a2c6eda0e0ad87b02eb43dcd0000ff495427532041205b4c4e445d205448494e47205f206e6f646c2023382f323100000701b0181b252607 malformed? (version v0.7.0-471-ge902d9a)
0x55f0f8376e39 send_backtrace
        common/daemon.c:40
0x55f0f837c598 status_failed
        common/status.c:192
0x55f0f836a3d6 process_pending_node_announcement
        gossipd/routing.c:1315
0x55f0f836b953 routing_add_channel_update
        gossipd/routing.c:1892
0x55f0f8366627 gossip_store_load
        gossipd/gossip_store.c:582
0x55f0f8371c7b gossip_init
        gossipd/gossipd.c:1862
0x55f0f8373f2e recv_req
        gossipd/gossipd.c:2702
0x55f0f8377343 handle_read
        common/daemon_conn.c:31
0x55f0f839a8cc next_plan
        ccan/ccan/io/io.c:59
0x55f0f839b449 do_plan
        ccan/ccan/io/io.c:407
0x55f0f839b487 io_ready
        ccan/ccan/io/io.c:417
0x55f0f839d4d6 io_loop
        ccan/ccan/io/poll.c:445
0x55f0f8374274 main
        gossipd/gossipd.c:2827
0x7ff3a6ee52e0 ???
        ???:0
0x55f0f8360db9 ???
        ???:0
0xffffffffffffffff ???
        ???:0
2019-06-07T13:13:43.787Z lightning_gossipd(20000): gossip_store node_announcement 10655004 replaces 10655004!
2019-06-07T13:13:43.788Z lightning_gossipd(20000): pending node_announcement 0101cd42b588499d54b9a95c48927f52951dd465dd9244f0cbb91a4035763a185a9e20c5b33bdf30ac67be2867faf4847367f1a4a65534c83d23fa124933b8993c1700005cf16af203cbea26f993aa7873c16798a161bc6cd418180c81a2c6eda0e0ad87b02eb43dcd0000ff495427532041205b4c4e445d205448494e47205f206e6f646c2023382f323100000701b0181b252607 malformed? (version v0.7.0-471-ge902d9a)
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/daemon.c:45 (send_backtrace) 0x55f0f8376e8f
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/status.c:192 (status_failed) 0x55f0f837c598
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/routing.c:1315 (process_pending_node_announcement) 0x55f0f836a3d6
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/routing.c:1892 (routing_add_channel_update) 0x55f0f836b953
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossip_store.c:582 (gossip_store_load) 0x55f0f8366627
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:1862 (gossip_init) 0x55f0f8371c7b
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:2702 (recv_req) 0x55f0f8373f2e
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/daemon_conn.c:31 (handle_read) 0x55f0f8377343
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55f0f839a8cc
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55f0f839b449
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55f0f839b487
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55f0f839d4d6
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:2827 (main) 0x55f0f8374274
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0x7ff3a6ee52e0
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0x55f0f8360db9
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-06-07T13:13:43.789Z lightning_gossipd(20000): STATUS_FAIL_INTERNAL_ERROR: pending node_announcement 0101cd42b588499d54b9a95c48927f52951dd465dd9244f0cbb91a4035763a185a9e20c5b33bdf30ac67be2867faf4847367f1a4a65534c83d23fa124933b8993c1700005cf16af203cbea26f993aa7873c16798a161bc6cd418180c81a2c6eda0e0ad87b02eb43dcd0000ff495427532041205b4c4e445d205448494e47205f206e6f646c2023382f323100000701b0181b252607 malformed?
lightningd: lightning_gossipd failed (exit status 242), exiting.

and this is the entire 2nd logfile:

2019-06-07T13:13:34.674Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_channeld
2019-06-07T13:13:34.702Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_closingd
2019-06-07T13:13:34.737Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_connectd
2019-06-07T13:13:34.755Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_gossipd
2019-06-07T13:13:34.772Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_hsmd
2019-06-07T13:13:34.798Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_onchaind
2019-06-07T13:13:34.819Z lightningd(19979): testing /usr/local/libexec/c-lightning/lightning_openingd
2019-06-07T13:13:41.448Z lightning_hsmd(19998): pid 19998, msgfd 17
2019-06-07T13:13:41.529Z lightning_connectd(19999): pid 19999, msgfd 20
2019-06-07T13:13:41.529Z lightning_hsmd(19998): Client: Received message 11 from client
2019-06-07T13:13:41.529Z lightning_hsmd(19998): Client: Received message 9 from client
2019-06-07T13:13:41.529Z lightning_hsmd(19998): new_client: 0
2019-06-07T13:13:41.606Z lightning_connectd(19999): Created IPv4 listener on port 9735
2019-06-07T13:13:41.606Z lightning_connectd(19999): REPLY WIRE_CONNECTCTL_INIT_REPLY with 0 fds
2019-06-07T13:13:41.607Z lightning_gossipd(20000): pid 20000, msgfd 19
2019-06-07T13:13:41.831Z lightning_hsmd(19998): Client: Received message 9 from client
2019-06-07T13:13:41.831Z lightning_hsmd(19998): new_client: 0
2019-06-07T13:13:41.851Z lightningd(19979): Smoothed feerate estimate for urgent initialized to polled estimate 19842
2019-06-07T13:13:41.851Z lightningd(19979): Feerate estimate for urgent set to 19842 (was 0)
2019-06-07T13:13:41.851Z lightningd(19979): Smoothed feerate estimate for normal initialized to polled estimate 13929
2019-06-07T13:13:41.851Z lightningd(19979): Feerate estimate for normal set to 13929 (was 0)
2019-06-07T13:13:41.851Z lightningd(19979): Smoothed feerate estimate for slow initialized to polled estimate 295
2019-06-07T13:13:41.851Z lightningd(19979): Feerate estimate for slow set to 295 (was 0)
2019-06-07T13:13:43.787Z lightning_gossipd(20000): gossip_store node_announcement 10655004 replaces 10655004!
2019-06-07T13:13:43.788Z lightning_gossipd(20000): pending node_announcement 0101cd42b588499d54b9a95c48927f52951dd465dd9244f0cbb91a4035763a185a9e20c5b33bdf30ac67be2867faf4847367f1a4a65534c83d23fa124933b8993c1700005cf16af203cbea26f993aa78
73c16798a161bc6cd418180c81a2c6eda0e0ad87b02eb43dcd0000ff495427532041205b4c4e445d205448494e47205f206e6f646c2023382f323100000701b0181b252607 malformed? (version v0.7.0-471-ge902d9a)
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/daemon.c:45 (send_backtrace) 0x55f0f8376e8f
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/status.c:192 (status_failed) 0x55f0f837c598
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/routing.c:1315 (process_pending_node_announcement) 0x55f0f836a3d6
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/routing.c:1892 (routing_add_channel_update) 0x55f0f836b953
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossip_store.c:582 (gossip_store_load) 0x55f0f8366627
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:1862 (gossip_init) 0x55f0f8371c7b
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:2702 (recv_req) 0x55f0f8373f2e
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: common/daemon_conn.c:31 (handle_read) 0x55f0f8377343
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55f0f839a8cc
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55f0f839b449
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55f0f839b487
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55f0f839d4d6
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: gossipd/gossipd.c:2827 (main) 0x55f0f8374274
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0x7ff3a6ee52e0
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0x55f0f8360db9
2019-06-07T13:13:43.789Z lightning_gossipd(20000): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-06-07T13:13:43.789Z lightning_gossipd(20000): STATUS_FAIL_INTERNAL_ERROR: pending node_announcement 0101cd42b588499d54b9a95c48927f52951dd465dd9244f0cbb91a4035763a185a9e20c5b33bdf30ac67be2867faf4847367f1a4a65534c83d23fa124933b8993c17
00005cf16af203cbea26f993aa7873c16798a161bc6cd418180c81a2c6eda0e0ad87b02eb43dcd0000ff495427532041205b4c4e445d205448494e47205f206e6f646c2023382f323100000701b0181b252607 malformed?

I can provide the complete 1st logfile if desired, but it is about 1 GB.

NOTE: During those3 days I did closed all 4 channels and opened 2 new ones (to get new announcement_signatures).

[thestick613]

I've deleted the gossip store, and now lightingd is able to start. However, listfunds doesn't show any outputs.

listpeers
{
   "peers" : []
}
listfunds
{
   "outputs" : [],
   "channels" : []
}

[m-schmoock]

I have the same issue on testnet. See my attached gossip_store that reproduces it:

gossip_store.zip

Stacktrace is:

2019-06-08T09:29:13.471Z lightningd(19468): --------------------------------------------------
2019-06-08T09:29:13.471Z lightningd(19468): Server started with public key 022368baffc9c5a46e7e0aa9690dd753cba2889e42498369c3643efbb422b492f9, alias dev.schmoock.net (color #ffdd00) and lightningd v0.7.0-521-gb6b548a
2019-06-08T09:29:13.472Z plugin-autoreload.py Forwarding {'jsonrpc': '2.0', 'method': 'init', 'params': {'options': {'cltv-final': '10', 'autoreload-plugin': '/home/will/projects/lightning.plugins.git/drain/drain.py'}, 'configuration': {'lightning-dir': '/home/will/.lightning', 'rpc-file': 'lightning-rpc'}}, 'id': 8}
2019-06-08T09:29:13.472Z plugin-autoreload.py Sent init to child plugin
2019-06-08T09:29:13.473Z plugin-rebalance.py Plugin rebalance.py initialized
2019-06-08T09:29:13.473Z plugin-summary.py Plugin summary.py initialized
2019-06-08T09:29:13.474Z plugin-autoreload.py Plugin drain.py initialized
2019-06-08T09:29:13.474Z plugin-sendinvoiceless.py Plugin sendinvoiceless.py initialized
2019-06-08T09:29:13.493Z lightning_gossipd(19496): gossip_store node_announcement 1635 replaces 1635!
2019-06-08T09:29:13.493Z lightning_gossipd(19496): pending node_announcement 0101fed21f0f12167f4410fff5705f4e109fe0f7f12fa6208da3b320e04fc5f3716205894c7ab1a203e49d7c68297c670255e0322719f6cd7356273333ae80e0e1a000005cf98add022368baffc9c5a46e7e0aa9690dd753cba2889e42498369c3643efbb422b492f9ffdd006465762e7363686d6f6f636b2e6e6574000000000000000000000000000000000000 malformed? (version v0.7.0-521-gb6b548a)
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: common/daemon.c:45 (send_backtrace) 0x55983116b839
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: common/status.c:192 (status_failed) 0x559831170fef
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: gossipd/routing.c:1315 (process_pending_node_announcement) 0x55983115ed6d
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: gossipd/routing.c:1891 (routing_add_channel_update) 0x5598311602ca
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: gossipd/gossip_store.c:582 (gossip_store_load) 0x55983115b0b3
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: gossipd/gossipd.c:1862 (gossip_init) 0x55983116661c
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: gossipd/gossipd.c:2702 (recv_req) 0x5598311688d1
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: common/daemon_conn.c:31 (handle_read) 0x55983116bced
2019-06-08T09:29:13.493Z lightning_gossipd(19496): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55983118f43b
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55983118ffb8
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55983118fff6
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55983119204b
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: gossipd/gossipd.c:2827 (main) 0x559831168c17
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: (null):0 ((null)) 0x7f4bd8fcbce2
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: (null):0 ((null)) 0x55983115585d
2019-06-08T09:29:13.494Z lightning_gossipd(19496): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-06-08T09:29:13.494Z lightning_gossipd(19496): STATUS_FAIL_INTERNAL_ERROR: pending node_announcement 0101fed21f0f12167f4410fff5705f4e109fe0f7f12fa6208da3b320e04fc5f3716205894c7ab1a203e49d7c68297c670255e0322719f6cd7356273333ae80e0e1a000005cf98add022368baffc9c5a46e7e0aa9690dd753cba2889e42498369c3643efbb422b492f9ffdd006465762e7363686d6f6f636b2e6e6574000000000000000000000000000000000000 malformed?
2019-06-08T09:29:13.494Z lightning_gossipd(19496): Status closed, but waitpid 19496 says No child processes

[darosior]

I've deleted the gossip store, and now lightingd is able to start. However, listfunds doesn't show any outputs.

For onchain funds you can dev-rescan-outputs Edit: You need to have compiled lightningd with ./configure --enable-developer

[thestick613]

Trouble is, i've already run dev-rescan-outputs, and the funds still aren't there.

[fixone]

I might have stumbled upon the same error (STATUS_FAIL_INTERNAL_ERROR: pending node_announcement) after the last compile

2019-06-09T05:50:47.227Z lightning_gossipd(28420): pending node_announcement 01017bb617c063668df79e6f5816d46a1a0d94b99b1acc11c3d5371815edba17cb4a1ed778126ec1c4ab9185813feab2be1625c4b9e17d724e60486d5a4bdff64edc00005cf7e13a033978dbf4a2a27aa68d5472f693ce5d96f57ca2866c068c59d7e42bd8462a65b900ff00496e506179546573746e65740000000000000000000000000000000000000000000701d42ff2532607 malformed? (version v0.7.0-525-g9e511cb)
2019-06-09T05:50:47.227Z lightning_gossipd(28420): backtrace: common/daemon.c:45 (send_backtrace) 0x5d0f9940696c
2019-06-09T05:50:47.228Z lightning_gossipd(28420): backtrace: common/status.c:192 (status_failed) 0x5d0f994094f8
2019-06-09T05:50:47.228Z lightning_gossipd(28420): backtrace: gossipd/routing.c:1315 (process_pending_node_announcement) 0x5d0f993febbc
2019-06-09T05:50:47.228Z lightning_gossipd(28420): backtrace: gossipd/routing.c:1891 (routing_add_channel_update) 0x5d0f993feeab
2019-06-09T05:50:47.228Z lightning_gossipd(28420): backtrace: gossipd/gossip_store.c:582 (gossip_store_load) 0x5d0f993fba94
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: gossipd/gossipd.c:1862 (gossip_init) 0x5d0f99403b84
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: gossipd/gossipd.c:2702 (recv_req) 0x5d0f99404b55
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: common/daemon_conn.c:31 (handle_read) 0x5d0f99406a78
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x5d0f9941d9b9
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x5d0f9941dea1
2019-06-09T05:50:47.229Z lightning_gossipd(28420): backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x5d0f9941dece
2019-06-09T05:50:47.230Z lightning_gossipd(28420): backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x5d0f9941f6a0
2019-06-09T05:50:47.230Z lightning_gossipd(28420): backtrace: gossipd/gossipd.c:2827 (main) 0x5d0f99404d4b
2019-06-09T05:50:47.230Z lightning_gossipd(28420): backtrace: (null):0 ((null)) 0x7d1f4bd062e0
2019-06-09T05:50:47.231Z lightning_gossipd(28420): backtrace: (null):0 ((null)) 0x5d0f993f7c19
2019-06-09T05:50:47.231Z lightning_gossipd(28420): backtrace: (null):0 ((null)) 0xffffffffffffffff
2019-06-09T05:50:47.231Z lightning_gossipd(28420): STATUS_FAIL_INTERNAL_ERROR: pending node_announcement 01017bb617c063668df79e6f5816d46a1a0d94b99b1acc11c3d5371815edba17cb4a1ed778126ec1c4ab9185813feab2be1625c4b9e17d724e60486d5a4bdff64edc00005cf7e13a033978dbf4a2a27aa68d5472f693ce5d96f57ca2866c068c59d7e42bd8462a65b900ff00496e506179546573746e65740000000000000000000000000000000000000000000701d42ff2532607 malformed?

Version is v0.7.0-525-g9e511cb