CLN terminating because of clnrest plugin

hMsats commented 6 months ago

Issue and Steps to Reproduce

2024-04-05T05:47:42.319Z UNUSUAL plugin-clnrest.py: Killing plugin: timed out before replying to getmanifest
2024-04-05T05:47:42.416Z UNUSUAL plugin-clnrest.py: Killing plugin: Invalid/missing result tok in '{"jsonrpc": "2.0","id": 16,"error": {"code":-4, "message":"Plugin terminated before replying to RPC call."}}'
2024-04-05T05:47:42.417Z **BROKEN** plugin-clnrest.py: Plugin marked as important, shutting down lightningd!
Can't recover from plugin failure, terminating.

I also don't understand why clnrest is marked as important while the documentation says: "You will need some Python packages if you want to use clnrest."

After again executing pip3 install --user flask-cors flask_restx pyln-client flask-socketio gevent gevent-websocket CLN started normally but I don't know if that has anything to do with it.

`getinfo` output

v24.02.1

ShahanaFarooqui commented 6 months ago

I also don't understand why clnrest is marked as important while the documentation says: "You will need some Python packages if you want to use clnrest."

Setting clnrest-port value in the config makes clnrest plugin important. If it is not set, then CLN will work properly without clnrest and its dependencies. But if the port is set and dependencies are missing then plugin will obviously fail. Core lightning prefers to fail in that event, so that user can either remove the 'clnrest-port` config option or fix the plugin dependencies.

After again executing pip3 install --user flask-cors flask_restx pyln-client flask-socketio gevent gevent-websocket CLN started normally but I don't know if that has anything to do with it.

The first UNUSUAL message and installation fix shows that there was some missing dependency. We cannot identify it without looking at logs from your initial execution of pip install or complete logs from Core Lightning when it was failing.

Closing the issue as no further action is required. Please feel free to comment if you have any more question.

hMsats commented 6 months ago

@ShahanaFarooqui

Setting clnrest-port value in the config makes clnrest plugin important.

I haven't set clnrest-port in my core lightning config file or do you mean something else?

ShahanaFarooqui commented 6 months ago

@hMsats That is strange. Do you have rest-port setting in your config without plugin c-lightning-REST? If not, can you share your config settings after masking sensitive information.

hMsats commented 6 months ago

@ShahanaFarooqui

This is my config-file and since there's no sensitive information in it anymore, I didn't have to mask anything:

bind-addr=192.168.178.20:9735
alias=bitcoinserver.nl
network=bitcoin
proxy=127.0.0.1:9050
fee-base=0
fee-per-satoshi=0
important-plugin=/home/user/plugins/backup/backup.py
experimental-offers
allow-deprecated-apis=false

I do have a plugins directory in my .lightning (don't remember if that's default) but it is empty.

I start CLN like this:

/home/user/lightning/lightningd/lightningd --lightning-dir=/media/ssd/.lightning

ShahanaFarooqui commented 6 months ago

Without clnrest-port and rest-port (old version), it is improbable for core lightning to fail on clnrest plugin. I am unable to reproduce the error with above config and environment.

hMsats commented 6 months ago

I had my CLN sometimes crashing because of the plugin-clnrest since half a year or so but got clnrest working (meaning got rid of the error message) by installing the necessary dependencies. It's only recently that I decided to create an issue about it. It's very strange indeed. Let's leave it here for the moment until another user reports the same issue. Thanks for looking into it and good luck developing clnrest further!

hMsats commented 6 months ago

@ShahanaFarooqui

One way to reproduce something similar is to uninstall flask_cors. Then with my config above, I get the message:

plugin-clnret.py: Killing plugin: disabled itself: No module named 'flask-cors'

ShahanaFarooqui commented 6 months ago

Ohh sorry, I forgot. We had this issue before for nix OS also.

The clnrest plugin checks for dependencies even before checking for plugin options and if it is missing, it will fail itself and core lightning as well. The solution to this issue is to explicitly disable the plugin in the config.

hMsats commented 6 months ago

Ah, glad this was useful. The obvious solution would be to check for plugin options before checking for dependencies otherwise clnrest and especially the python dependencies are not optional anymore (unless you disable it explicitly but who will know about that and it goes against the ethos of plugins I assume). However, changing that order will probably create other difficulties, so that's for you developers to decide. This still leaves the problem that clnrest is treated as important even when it's never specified in the config file, as in my original post.

rustyrussell commented 6 months ago

@ShahanaFarooqui

One way to reproduce something similar is to uninstall flask_cors. Then with my config above, I get the message:

plugin-clnret.py: Killing plugin: disabled itself: No module named 'flask-cors'

That's what is supposed to happen! But CLN should continue...

I'm not sure what this actual bug is! Perhaps your error is something other than a missing module? If so, we would expect to see something in stderr from CLN (which is not captured, deliberately).

hMsats commented 6 months ago

@rustyrussell It does continue when flask-cors isn't present but it did not continue in my original issue. But apart from that, it shouldn't start clnrest in the first place because it's never specified in my config file. This makes clnrest opt out instead of opt in.

ShahanaFarooqui commented 6 months ago

Adding clnrest plugin's logical failure/disabling scenarios to clarify confusion in this issue and for posterity:

Missing Python: Till release v24.02, if python itself is missing then clnrest will fail and kill Core lightning as well (important plugin). The solution was to explicitly disable clnrest. Now, it has been fixed with commit to gracefully disable clnrest plugin if python is missing and Core lightning will work as expected.
Missing python library: It should gracefully disable the clnrest plugin only. Core Lightning will not be impacted.
If clnrest-port is not configured then the plugin will not start. So clnrest is actually opt-in not opt-out.

The original issue can't be related to missing python library or default opt-in.

ElementsProject / lightning