search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.84k stars 901 forks source link

Attempting single payment invoice crashes pay, clightning #4371

Closed 3nprob closed 3 years ago

3nprob commented 3 years ago

Issue and Steps to Reproduce

Attempting to pay a single payment receipt of 0.000005 btc (lnpay.co)

Outgoing capacity is ~0.06 BTC spread over several channels.

plugin-pay: cmd 154637 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat
pay: channel_update REDACTED ignored: fee 1000/1500000 cltv 50 too large
pay: channel_update REDACTED ignored: fee 100000/100000000 cltv 144 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 1000000/10000000 cltv 40 too large
pay: channel_update REDACTED ignored: fee 2950499017/422752876 cltv 34 too large
pay: channel_update REDACTED ignored: fee 749641351/722800720 cltv 34 too large
pay: channel_update REDACTED ignored: fee 1000/1500000 cltv 50 too large
pay: channel_update REDACTED ignored: fee 100000/100000000 cltv 144 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 1000000/10000000 cltv 40 too large
pay: channel_update REDACTED ignored: fee 2950499017/422752876 cltv 34 too large
pay: channel_update REDACTED ignored: fee 749641351/722800720 cltv 34 too large
pay: FATAL SIGNAL 11 (version v0.9.3)
0x558150cb817a send_backtrace
    common/daemon.c:38
0x558150cb8204 crashdump
    common/daemon.c:51
0x7f2fb705172f ???
    ???:0
0x558150cd5648 gossmap_chan_scid
    common/gossmap.c:555
0x558150ca8bc5 route_hops_from_route
    plugins/libplugin-pay.c:700
0x558150caadb3 payment_getroute
    plugins/libplugin-pay.c:807
0x558150cab10f payment_continue
    plugins/libplugin-pay.c:1903
0x558150cad8a2 adaptive_splitter_cb
    plugins/libplugin-pay.c:3486
0x558150cab0e1 payment_continue
    plugins/libplugin-pay.c:1895
0x558150cacb8c retry_step_cb
    plugins/libplugin-pay.c:2045
0x558150cab0e1 payment_continue
    plugins/libplugin-pay.c:1895
0x558150cabda5 waitblockheight_cb
    plugins/libplugin-pay.c:3054
0x558150cab0e1 payment_continue
    plugins/libplugin-pay.c:1895
0x558150cac8c4 presplit_cb
    plugins/libplugin-pay.c:3279
0x558150cab0e1 payment_continue
    plugins/libplugin-pay.c:1895
0x558150cabe82 payee_incoming_limit_count
    plugins/libplugin-pay.c:3570
0x558150ca61cb handle_rpc_reply
    plugins/libplugin.c:564
0x558150ca62bd rpc_read_response_one
    plugins/libplugin.c:711
0x558150ca6373 rpc_conn_read_response
    plugins/libplugin.c:731
0x558150cc90b3 next_plan
    ccan/ccan/io/io.c:59
0x558150cc9586 do_plan
    ccan/ccan/io/io.c:407
0x558150cc95b8 io_ready
    ccan/ccan/io/io.c:417
0x558150ccae70 io_loop
    ccan/ccan/io/poll.c:445
0x558150ca6e9c plugin_main
    plugins/libplugin.c:1414
0x558150ca3a16 main
    plugins/pay.c:2175
0x7f2fb6e3809a ???
    ???:0
0x558150c9edf9 ???
    ???:0
0xffffffffffffffff ???
    ???:0

getinfo output

{
   "num_peers": 16,
   "num_pending_channels": 0,
   "num_active_channels": 9,
   "num_inactive_channels": 0,
   "blockheight": 669067,
   "network": "bitcoin",
   "msatoshi_fees_collected": 0,
   "fees_collected_msat": "0msat",
}
3nprob commented 3 years ago

paystatus:

         "attempts": [
            {
               "strategy": "Initial attempt",
               "start_time": "2021-02-04T14:39:46.865Z",
               "age_in_seconds": 910,
               "end_time": "2021-02-04T14:45:01.777Z",
               "failure": {
                  "code": 205,
                  "message": "Call to getroute: Could not find a route"
               }
            }
         ]

There is a 2-hop path from me to the destination.

Calling getroute ID 500000 RISK yields path of length 2 - 5 depending on the value for RISK

cdecker commented 3 years ago

Can you share the version (commit hash) that this is running? I'm having a bit of trouble finding the cause since the line numbers in the backtrace don't match up. Assigned @rustyrussell since gossmap is his baby :-)

3nprob commented 3 years ago

I just tried making a payment again to another invoice/node.

(I did rebuild between the attempts but I'm 99% sure it was the same commit hash up above as well)

I'm running version: v0.9.3 (015ac37d924e31bb87b8597da9f863e82270657b)

This time, nothing shows up in paystatus anymore.

stderr from this attempt:

pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 4109209170/1237590906 cltv 34 too large
pay: channel_update REDACTED ignored: fee 1484438641/3286469617 cltv 34 too large
pay: channel_update REDACTED ignored: fee 1000000/10000000 cltv 40 too large
pay: channel_update REDACTED ignored: fee 100000/100000000 cltv 144 too large
pay: channel_update REDACTED ignored: fee 1000/1500000 cltv 50 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 7/6000000 cltv 4 too large
pay: channel_update REDACTED ignored: fee 4109209170/1237590906 cltv 34 too large
pay: channel_update REDACTED ignored: fee 1484438641/3286469617 cltv 34 too large
pay: channel_update REDACTED ignored: fee 1000000/10000000 cltv 40 too large
pay: channel_update REDACTED ignored: fee 100000/100000000 cltv 144 too large
pay: channel_update REDACTED ignored: fee 1000/1500000 cltv 50 too large
pay: FATAL SIGNAL 11 (version v0.9.3)
0x55abf43c517a send_backtrace
        common/daemon.c:38
0x55abf43c5204 crashdump
        common/daemon.c:51
0x7f6ff767b72f ???
        ???:0
0x55abf43e2648 gossmap_chan_scid
        common/gossmap.c:555
0x55abf43b5bc5 route_hops_from_route
        plugins/libplugin-pay.c:700
0x55abf43b7db3 payment_getroute
        plugins/libplugin-pay.c:807
0x55abf43b810f payment_continue
        plugins/libplugin-pay.c:1903
0x55abf43ba8a2 adaptive_splitter_cb
        plugins/libplugin-pay.c:3486
0x55abf43b80e1 payment_continue
        plugins/libplugin-pay.c:1895
0x55abf43b9b8c retry_step_cb
        plugins/libplugin-pay.c:2045
0x55abf43b80e1 payment_continue
        plugins/libplugin-pay.c:1895
0x55abf43b8da5 waitblockheight_cb
        plugins/libplugin-pay.c:3054
0x55abf43b80e1 payment_continue
        plugins/libplugin-pay.c:1895
0x55abf43b98c4 presplit_cb
        plugins/libplugin-pay.c:3279
0x55abf43b80e1 payment_continue
        plugins/libplugin-pay.c:1895
0x55abf43b8e82 payee_incoming_limit_count
        plugins/libplugin-pay.c:3570
0x55abf43b31cb handle_rpc_reply
        plugins/libplugin.c:564
0x55abf43b32bd rpc_read_response_one
        plugins/libplugin.c:711
0x55abf43b3373 rpc_conn_read_response
        plugins/libplugin.c:731
0x55abf43d60b3 next_plan
        ccan/ccan/io/io.c:59
0x55abf43d6586 do_plan
        ccan/ccan/io/io.c:407
0x55abf43d65b8 io_ready
        ccan/ccan/io/io.c:417
0x55abf43d7e70 io_loop
        ccan/ccan/io/poll.c:445
0x55abf43b3e9c plugin_main
        plugins/libplugin.c:1414
0x55abf43b0a16 main
        plugins/pay.c:2175
0x7f6ff746209a ???
        ???:0
0x55abf43abdf9 ???
        ???:0
0xffffffffffffffff ???
        ???:0

stdout:

INFO    plugin-pay: cmd 73881 partid 0: Payment fee constraint 50msat is below exemption threshold, allowing a maximum fee of 5000msat
INFO    plugin-pay: Killing plugin: Plugin exited before completing handshake.
**BROKEN** plugin-pay: Plugin marked as important, shutting down lightningd!
cdecker commented 3 years ago

Thanks that makes it easier to track down. The null-pointer is here:

https://github.com/ElementsProject/lightning/blob/015ac37d924e31bb87b8597da9f863e82270657b/common/gossmap.c#L555

Called here:

https://github.com/ElementsProject/lightning/blob/015ac37d924e31bb87b8597da9f863e82270657b/plugins/libplugin-pay.c#L697-L711

Looks like r[i]->c is NULL, how could that be @rustyrussell?

3nprob commented 3 years ago

Since I see mention of onion in the surrounding code, I could add that this is a tor-only node (always-use-proxy=true)

3nprob commented 3 years ago

After the last payment, lightningd is now crashing intermittently (~once per 10-40 min). same stacktrace as above.

cdecker commented 3 years ago

Since I see mention of onion in the surrounding code, I could add that this is a tor-only node (always-use-proxy=true)

Nope, it's a different type of onion routing, not the transport level onion.

3nprob commented 3 years ago

My client is still crashlooping - any pointers on how I can troubleshoot further?

3nprob commented 3 years ago

It stopped crashing now - not sure what did it.New issue now though, but treating that as separate from this.