search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.85k stars 902 forks source link

lightningd crashes because of log messages #4385

Closed sgeisler closed 3 years ago

sgeisler commented 3 years ago

Issue and Steps to Reproduce

I noticed that lightningd was crashing quite often recently and it felt like it was getting worse over time. It always happens right after some error complaining about a malformed JSON reply which obviously is a log message, e.g.:

Feb 14 23:33:58 [redacted] lightningd[10539]: offers: Malformed JSON reply '2021-02-14T22:33:58.047Z DEBUG   hsmd: Client: Received message 10 from client

or after turning off debug output 

Feb 14 23:34:58 [redacted] lightningd[11514]: offers: Malformed JSON reply '2021-02-14T22:34:58.361Z INFO    plugin-fetchinvoice: Killing plugin: offers not enabled in config

Directly following it there's also a json dump of node config I assume, I can provide that privately if that helps.

After changing the log level to log-level=warn it seems to work better now (ran for 15min without crashing).

getinfo output

Not doxing myself here, but can of course send it, although I don't think it would help in this case.

I'm running v0.9.3 (015ac37d924e31bb87b8597da9f863e82270657b) on debian 10.8 and disabled all plugins for debugging.

cdecker commented 3 years ago

This is a known issue if you do not provide a --log-file and --daemon. We close the stdout file-descriptor which gets reused for plugins, leading to this issue.

It has been addressed in master: https://github.com/ElementsProject/lightning/pull/4350

sgeisler commented 3 years ago

Oh, sorry, somehow I didn't find that issue :disappointed: nice to see it's already fixed! :smiley: