sauron: c-lightning crashes while downloading blocks

[kiwiidb]

I tried to get c-lightning to work using the sauron plugin, but it keeps crashing from time to time with the following logs:

...
2021-02-15T18:36:14.319Z DEBUG   lightningd: Adding block 670729: 0000000000000000000484c76d9d54ea06b1758c7852df8f516e177a9cf9ca9f
lightningd: FATAL SIGNAL 11 (version v0.9.3)
0x55bd5ded77f6 send_backtrace
    common/daemon.c:38
0x55bd5ded7880 crashdump
    common/daemon.c:51
0x7f621745472f ???
    ???:0
0x55bd5dee870e bitcoin_block_from_hex
    bitcoin/block.c:214
0x55bd5de8fb93 getrawblockbyheight_callback
    lightningd/bitcoind.c:436
0x55bd5debaef8 plugin_response_handle
    lightningd/plugin.c:432
0x55bd5debb01a plugin_read_json_one
    lightningd/plugin.c:538
0x55bd5debb19a plugin_read_json
    lightningd/plugin.c:583
0x55bd5df0fda4 next_plan
    ccan/ccan/io/io.c:59
0x55bd5df10277 do_plan
    ccan/ccan/io/io.c:407
0x55bd5df102a9 io_ready
    ccan/ccan/io/io.c:417
0x55bd5df11b61 io_loop
    ccan/ccan/io/poll.c:445
0x55bd5de9e504 io_loop_with_timers
    lightningd/io_loop_with_timers.c:24
0x55bd5dea1d75 main
    lightningd/lightningd.c:1016
0x7f621728609a ???
    ???:0
0x55bd5de8f1f9 ???
    ???:0
0xffffffffffffffff ???
    ???:0
2021-02-15T18:36:50.306Z **BROKEN** lightningd: FATAL SIGNAL 11 (version v0.9.3)
2021-02-15T18:36:50.306Z **BROKEN** lightningd: backtrace: common/daemon.c:43 (send_backtrace) 0x55bd5ded783e
2021-02-15T18:36:50.306Z **BROKEN** lightningd: backtrace: common/daemon.c:51 (crashdump) 0x55bd5ded7880
2021-02-15T18:36:50.306Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f621745472f
2021-02-15T18:36:50.306Z **BROKEN** lightningd: backtrace: bitcoin/block.c:214 (bitcoin_block_from_hex) 0x55bd5dee870e
2021-02-15T18:36:50.306Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:436 (getrawblockbyheight_callback) 0x55bd5de8fb93
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:432 (plugin_response_handle) 0x55bd5debaef8
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:538 (plugin_read_json_one) 0x55bd5debb01a
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:583 (plugin_read_json) 0x55bd5debb19a
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x55bd5df0fda4
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x55bd5df10277
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x55bd5df102a9
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:445 (io_loop) 0x55bd5df11b61
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:24 (io_loop_with_timers) 0x55bd5de9e504
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1016 (main) 0x55bd5dea1d75
2021-02-15T18:36:50.307Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f621728609a
2021-02-15T18:36:50.308Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x55bd5de8f1f9
2021-02-15T18:36:50.308Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
Log dumped in crash.log.20210215183650
spenderp: Lost connection to the RPC socket.
txprepare: Lost connection to the RPC socket.
pay: Lost connection to the RPC socket.
keysend: Lost connection to the RPC socket.

I am using the Dockerfile from this repository (with a small tweak to make it work ) , and run it with: docker run lightningd --disable-plugin bcli --plugin /opt/lightningd/plugins/sauron/sauron.py --sauron-api-endpoint https://blockstream.info/api --log-level debug It seems that after a while, when c-lightning is up-to-data with the chain tip, there are no more crashes.

[cdecker]

This looks like Sauron may end up in some cases not returning a valid json string, for example this could happen if the download fails for some reason. Sauron should likely catch these cases and retry after a bit, or use another backend until the original backend returns to normal.

[f-hayek]

Running sauron like this:

lightningd --mainnet --disable-plugin bcli --plugin $PWD/sauron/sauron.py --sauron-api-endpoint https://blockstream.info/api/

After running for a while it crashes with:

/home/cln/plugins/sauron/sauron.py error: bad response to getutxout (bad 'result' field: Parsing '{result:': object does not have member result), response was {"jsonrpc": "2.0", "id": 470, "error": {"code": -32600, "message": "Error while processing getutxout: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))"

[ca-ruz]

It looks like @darosior fixed this in https://github.com/lightningd/plugins/pull/297.

[chrisguida]

Thanks @ca-ruz ! This indeed seems to be fixed now. Feel free to reopen if it's not.