Missing result from listincoming. No successfull start.

[Schnema1]

Issue:

CLN 24.08rc3 The OS ran out of disc space and core lightning crashed. After extending the LVM and system reboot core lighting did not start anymore.

Measures: I then tried to read the logs and found an issue similair to the error messages I got. So i deleted the gossip_store* files. This did not help and I decided to compile version 24.08. Again, no boot. Disabling plugins does not help, nor does disabling cln-rest helps.

getinfo output

Long break:

Line 192 and 193 it takes a long time to compact the (gossip store?) 2024-09-04T09:53:23.255Z DEBUG hsmd: new_client: 0 2024-09-04T09:56:31.128Z DEBUG gossipd: Store compact time: 187252 msec

Maybe the error occurs on line 441+

2024-09-04T09:56:32.200Z DEBUG plugin-clnrest: Notification: {'warning': {'level': 'warn', 'time': '1725443792.198330200', 'timestamp': '2024-094T09:56:32.198Z', 'source': 'chan#86', 'log': 'Replaying old unprocessed HTLC #12574'}} 2024-09-04T09:56:32.233Z DEBUG lightningd: Feerate estimate for 100 blocks set to 605 (was 606) 2024-09-04T09:56:32.389Z DEBUG lightningd: Adding block 856261: 000000000000000000006a6a5ef320255c7889b28f4af0501cd75d2fb3729d64 2024-09-04T09:56:34.358Z INFO plugin-topology: Killing plugin: exited during normal operation 2024-09-04T09:56:34.358Z DEBUG lightningd: routehint: {"jsonrpc": "2.0","id": "cln:htlc_accepted#42/keysend:invoice#3/cln:listincoming#43","erro: {"code":-4, "message":"Plugin terminated before replying to RPC call."}} 2024-09-04T09:56:34.358Z BROKEN lightningd: Missing result from listincoming: {"jsonrpc": "2.0","id": "cln:htlc_accepted#42/keysend:invoice#3/n:listincoming#43","error": {"code":-4, "message":"Plugin terminated before replying to RPC call."}} 2024-09-04T09:56:34.374Z BROKEN lightningd: FATAL SIGNAL 6 (version v24.08)

How can I fix this issue? debug.log

I added the whole DEBUG output in a file.

[Schnema1]

Ok, for anyone stumbling over this post, here is the fix.

First of all don't panic (although I did somehow)!

Jump in your config file and switch you log level to debug. Start lightningd and check tail - f or tail - 500 ./lighting/lighting.log if there are any open HTLC within a channel. If not, then you got time. See attached log line 66 to 97 and 156 to 190.

I have tried the measures mentioned in first post with no success. I also tried to delete the gossip file. However, I navigated to /HOME/lightning/gossipd/ and found several gossip_store files. I deleted them without any success. < --- this is not good, let it be.

Later on I found in the Telegram group a post from Christian Decker mentioning the correct gossip_store path. It is in the private lighting dir. /HOME/.lightning/bitcoin/gossip_store

There have been also a gossip_store_corrupted. I then deleted both files and finally core lighting started successfully.

Tipp: Check out the crash.log files in this directory too. If you have a service file, so probably lighting will restart 30 or 60 seconds after a crash. So every approx 5 minutes you will get a crash log. Do not underestimate the size of these logs, especially if you are still in debug mode.

Tags:

INFO plugin-topology: Killing plugin: exited during normal operation

BROKEN lightningd: Missing result from listincoming: {"not relevant for this problem"}

gossip_store (HOME/.lighting/bitcoin/gossip_store*)