rebalancing (circular payment), crash c-lightning

[PsySc0rpi0n]

Issue and Steps to Reproduce

I have one channel that needs to be rebalanced. The balance is now around 15% / 85% (Out/In) and the total capacity is around 1.5 million sats. 1.4 million sats is now inbound capacity. So I tried to create an invoice with 985k sats (because I have another channel with 100% Incoming capacity of 1 million sats, minus reserves and fees) to try to send them through this channel and the logs says the follwoing:

2021-01-20T23:33:14.730Z DEBUG gossipd: Trying to find a route from (me) to 03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91 for 1000msat 2021-01-20T23:33:14.730Z INFO gossipd: find_route: this is 03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91, refusing to create empty route 2021-01-20T23:33:14.730Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GETROUTE_REPLY with 0 fds 2021-01-20T23:33:14.731Z DEBUG plugin-pay: cmd 36918 partid 0: Not using a routehint 2021-01-20T23:33:14.731Z DEBUG plugin-pay: cmd 36918 partid 0: The destination is not directly reachable excluding attempts without routehints 2021-01-20T23:33:14.731Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GETCHANNELS_REPLY with 0 fds 2021-01-20T23:33:14.732Z INFO plugin-pay: cmd 36918 partid 0: Initial limit on max HTLCs: 90, Destination 03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91 has 6 channels, assuming 15 HTLCs per channel 2021-01-20T23:33:14.732Z INFO plugin-pay: cmd 36918 partid 1: Split into 6 sub-payments due to initial size (985000000msat > 160000000msat): new partid 2, new partid 3, new partid 4, new partid 5, new partid 6, new partid 7 2021-01-20T23:33:14.733Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GETCHANNELS_REPLY with 0 fds pay: channel_update 576683x1919x0 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 577477x1902x1 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 577480x1296x0 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 577480x1297x0 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 577555x1983x0 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 586709x610x0 ignored: fee 100000/100000000 cltv 144 too large pay: channel_update 660870x1328x0 ignored: fee 1000/1500000 cltv 50 too large pay: channel_update 665892x421x0 ignored: fee 1000/10000000 cltv 40 too large pay: channel_update 666486x2306x0 ignored: fee 1000/10000000 cltv 40 too large pay: channel_update 577590x1455x0 ignored: fee 7/6000000 cltv 4 too large pay: channel_update 666853x820x0 ignored: fee 1000/10000000 cltv 40 too large pay: channel_update 665401x1883x1 ignored: fee 1000/10000000 cltv 40 too large pay: channel_update 666900x1026x1 ignored: fee 1000/10000000 cltv 40 too large pay: channel_update 666529x884x0 ignored: fee 1000/10000000 cltv 40 too large pay: FATAL SIGNAL 6 (version v0.9.2-67-gff88308) 0x55d40d2222c7 send_backtrace common/daemon.c:38 0x55d40d22236d crashdump common/daemon.c:51 0x7fd88dcb483f ??? ???:0 0x7fd88dcb47bb ??? ???:0 0x7fd88dc9f534 ??? ???:0 0x55d40d20a4e8 payment_route_fee plugins/libplugin-pay.c:426 0x55d40d20b5cb payment_getroute plugins/libplugin-pay.c:807 0x55d40d20e730 payment_continue plugins/libplugin-pay.c:1897 0x55d40d212674 adaptive_splitter_cb plugins/libplugin-pay.c:3461 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d20eb81 retry_step_cb plugins/libplugin-pay.c:2039 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d2116dd waitblockheight_cb plugins/libplugin-pay.c:3031 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d211b1f presplit_cb plugins/libplugin-pay.c:3198 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d2127f8 payee_incoming_limit_step_cb plugins/libplugin-pay.c:3556 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d2103d0 routehint_step_cb plugins/libplugin-pay.c:2554 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d211084 shadow_route_cb plugins/libplugin-pay.c:2878 0x55d40d20e6e6 payment_continue plugins/libplugin-pay.c:1889 0x55d40d2110ed direct_pay_override plugins/libplugin-pay.c:2899 0x55d40d21146d direct_pay_listpeers plugins/libplugin-pay.c:2963 0x55d40d205117 handle_rpc_reply plugins/libplugin.c:560 0x55d40d205a80 rpc_read_response_one plugins/libplugin.c:707 0x55d40d205b9d rpc_conn_read_response plugins/libplugin.c:727 0x55d40d23a0dc next_plan ccan/ccan/io/io.c:59 0x55d40d23ac59 do_plan ccan/ccan/io/io.c:407 0x55d40d23ac97 io_ready ccan/ccan/io/io.c:417 0x55d40d23ce5d io_loop ccan/ccan/io/poll.c:445 0x55d40d20811c plugin_main plugins/libplugin.c:1403 0x55d40d203c8c main plugins/pay.c:2103 0x7fd88dca109a ??? ???:0 0x55d40d1fcd99 ??? ???:0 0xffffffffffffffff ??? ???:0 pay: FATAL SIGNAL 11 (version v0.9.2-67-gff88308) 0x55d40d2222c7 send_backtrace common/daemon.c:38 0x55d40d22236d crashdump common/daemon.c:51 0x7fd88dcb483f ??? ???:0 0x0 ??? ???:0 2021-01-20T23:33:14.899Z INFO plugin-pay: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.899Z BROKEN plugin-pay: Plugin marked as important, shutting down lightningd! 2021-01-20T23:33:14.900Z DEBUG connectd: Shutting down 2021-01-20T23:33:14.914Z DEBUG 02ab6d60e2e46e1af91c671234c914f99a3826c1c7e3a358cc46a0bcf37663adb2-channeld-chan#10: Status closed, but not exited. Killing 2021-01-20T23:33:14.915Z DEBUG 03e691f81f08c56fa876cc4ef5c9e8b727bd682cf35605be25d48607a802526053-channeld-chan#18: Status closed, but not exited. Killing 2021-01-20T23:33:14.916Z DEBUG 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-channeld-chan#37: Status closed, but not exited. Killing 2021-01-20T23:33:14.916Z DEBUG 0239e4dc85543cd84b712850072dc5e1bf0190c44b64dbe822581013f849a794a4-channeld-chan#46: Status closed, but not exited. Killing 2021-01-20T23:33:14.917Z DEBUG 03691d4dc8789455bf9241b9f808aaca92099851b1db92f4031e2ae997f3eea711-channeld-chan#47: Status closed, but not exited. Killing 2021-01-20T23:33:14.918Z DEBUG 03e9e5a2ae70e42d012c2cecc5a2662630b97efeb13c4080168e19f23a518051c5-openingd-chan#48: Status closed, but not exited. Killing 2021-01-20T23:33:14.918Z DEBUG 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b-openingd-chan#50: Status closed, but not exited. Killing 2021-01-20T23:33:14.919Z DEBUG 03968d7f8825d1e34e7d38993617cb6a861f7bcf78e6fe1b1043272350960d1f51-channeld-chan#58: Status closed, but not exited. Killing 2021-01-20T23:33:14.920Z DEBUG 032cc4541b25e86e39a7d450a979c1a9adbe2878df3a93fcb59c96c700bfe26aa3-openingd-chan#64: Status closed, but not exited. Killing 2021-01-20T23:33:14.920Z DEBUG 03295d2e292565743a40bd44da227a820f8730877bc3dfadebade8785bcf355258-openingd-chan#65: Status closed, but not exited. Killing 2021-01-20T23:33:14.921Z DEBUG gossipd: Shutting down 2021-01-20T23:33:14.937Z DEBUG hsmd: Shutting down 2021-01-20T23:33:14.937Z INFO plugin-autoclean: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.937Z INFO plugin-bcli: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.937Z INFO plugin-keysend: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.937Z INFO plugin-txprepare: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.937Z INFO plugin-spenderp: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.937Z INFO plugin-backup.py: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.938Z INFO plugin-summary.py: Killing plugin: Plugin exited before completing handshake. 2021-01-20T23:33:14.938Z INFO plugin-backup.py: Killing plugin: Plugin exited before completing handshake.

getinfo output

{ "id": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91", "alias": "CryptoAnarchMoV", "color": "03fef7", "num_peers": 6, "num_pending_channels": 0, "num_active_channels": 6, "num_inactive_channels": 0, "address": [ { "type": "torv3", "address": "isvcxqviqbhrzpzpcmeh6u5ve42pvcjdukckec4mhdcwhnkff6qexdqd.onion", "port": 9735 } ], "binding": [ { "type": "ipv4", "address": "127.0.0.1", "port": 9735 } ], "version": "v0.9.2-67-gff88308", "blockheight": 666962, "network": "bitcoin", "msatoshi_fees_collected": 25005, "fees_collected_msat": "25005msat", "lightning-dir": "/home/psysc0rpi0n/.lightning/bitcoin" }

Channel needing rebalancing:

{ "channels": [ { "source": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2", "destination": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91", "short_channel_id": "665969x1380x0", "public": true, "satoshis": 1575817, "amount_msat": "1575817000msat", "message_flags": 1, "channel_flags": 0, "active": true, "last_update": 1610884769, "base_fee_millisatoshi": 1000, "fee_per_millionth": 1, "delay": 40, "htlc_minimum_msat": "1000msat", "htlc_maximum_msat": "1575817000msat", "features": "" }, { "source": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91", "destination": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2", "short_channel_id": "665969x1380x0", "public": true, "satoshis": 1575817, "amount_msat": "1575817000msat", "message_flags": 1, "channel_flags": 1, "active": true, "last_update": 1611098741, "base_fee_millisatoshi": 1000, "fee_per_millionth": 250, "delay": 34, "htlc_minimum_msat": "1msat", "htlc_maximum_msat": "1560059000msat", "features": "" } ] }

Channel I was trying to use to send the funds for the rebalancing:

{ "channels": [ { "source": "03968d7f8825d1e34e7d38993617cb6a861f7bcf78e6fe1b1043272350960d1f51", "destination": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91", "short_channel_id": "666671x1102x0", "public": true, "satoshis": 1000000, "amount_msat": "1000000000msat", "message_flags": 1, "channel_flags": 0, "active": true, "last_update": 1611014880, "base_fee_millisatoshi": 1000, "fee_per_millionth": 1, "delay": 40, "htlc_minimum_msat": "1000msat", "htlc_maximum_msat": "1000000000msat", "features": "" }, { "source": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91", "destination": "03968d7f8825d1e34e7d38993617cb6a861f7bcf78e6fe1b1043272350960d1f51", "short_channel_id": "666671x1102x0", "public": true, "satoshis": 1000000, "amount_msat": "1000000000msat", "message_flags": 1, "channel_flags": 1, "active": true, "last_update": 1611098672, "base_fee_millisatoshi": 1000, "fee_per_millionth": 250, "delay": 34, "htlc_minimum_msat": "1msat", "htlc_maximum_msat": "990000000msat", "features": "" } ] }

The invoice I created:

$ lightning-cli invoice 985000000msat \"revalance-boltz-01\" \"Rebalancing_boltx_cahnnel\" { "payment_hash": "2e3678e78eccc3be3d5061a8bdc21b503a28f8faa89bff8df0da026adfbfca8f", "expires_at": 1611790349, "bolt11": "lnbc9850u1psq30vdpp59cm83euwenpmu02svx5tmssm2qaz37864zdllr0smgpx4hale28sdpg2fjkyctvv9hxx6twva0kymmvw3u97cmpdphxuetvxqyjw5qcqpjsp5d2lyyyd0eq2977lnq0f522yckhc2duexaxpzpzgycqekwwqf7dgsrzjqfsktpgyjffp7jkg40vmmqygzg6yd5fx7eyv5d0xp7ypwlwpf88tyz3fwyqq2eqqqqqqqqlgqqqqqqgq9q9qy9qsq5az79qzelwllnmtkf4l8pg8xkhelpxmj6z9tcppd2rv0yygx508k757arj84t0ar9rysm4yvwgxfyqyllkgr965llp264ql5hlz586cph60xlx" }

[ZmnSCPxj]

Note that we do not actually support self-paying via pay... previous versions did an early-out success (we are paying ourself, so the most logical thing to do is... not spend on fees to send a payment to ourself), probably we need to reimplement that. Instead, consider something from plugins or look into CLBOSS for automatic management.

The crashing line is:

https://github.com/ElementsProject/lightning/blob/ff8830876dc2ead010dc4a6453abb7e30ec637ce/plugins/libplugin-pay.c#L426

Which occurs because we are apparently generating a route that gives negative fees:

https://github.com/ElementsProject/lightning/blob/ff8830876dc2ead010dc4a6453abb7e30ec637ce/plugins/libplugin-pay.c#L417-L427

[ZmnSCPxj]

Below is the decodepay of your invoice:

{
   "currency": "bc",
   "created_at": 1611185549,
   "expiry": 604800,
   "payee": "03fef777d58a529df02a3fb267690e0c9033767b555cc1c63844bb2d3498789f91",
   "msatoshi": 985000000,
   "amount_msat": "985000000msat",
   "description": "Rebalancing_boltx_cahnnel",
   "min_final_cltv_expiry": 18,
   "payment_secret": "6abe4211afc8145f7bf303d3452898b5f0a6f326e982208904c033673809f351",
   "features": "028200",
   "routes": [
      [
         {
            "pubkey": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2",
            "short_channel_id": "665969x1380x0",
            "fee_base_msat": 1000,
            "fee_proportional_millionths": 1,
            "cltv_expiry_delta": 40
         }
      ]
   ],
   "payment_hash": "2e3678e78eccc3be3d5061a8bdc21b503a28f8faa89bff8df0da026adfbfca8f",
   "signature": "3045022100a745e28059fbbff9ed764d7e70a0e6b5f3f09b72d08abc042d50d8f21106a3cf02206f53dd1c8f55bfa328c90dd48c720c92009ffd9032ea9ff855aa83f4bfc543eb"
}

So there is a routehint from 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 to your node, which makes sense. Not sure what is causing the negative fees though.

[PsySc0rpi0n]

@ZmnSCPxj, @darosior told me in IRC that self-payments are supported... Do we or do we not support them? I'm confused.

[cdecker]

Strictly speaking we support paying to ourselves, we just won't create a circular route for it in pay / gossipd, which I think leads to a 0-hop route, which then crashes in the pay plugin. I'll take a look at it.

[cdecker]

Yes, a 0-hop route will end up failing here:

https://github.com/ElementsProject/lightning/blob/04f139f092324a2cbfbdf2c17b6cfd0cd687834d/plugins/libplugin-pay.c#L419

[PsySc0rpi0n]

@cdecker, so for now, it will not be possible to perform the rebalancing using this circular payment technique for now, right?

[whitslack]

@PsySc0rpi0n: You might want to use the rebalance plugin rather than attempting to implement rebalancing yourself manually.

[PsySc0rpi0n]

@PsySc0rpi0n: You might want to use the rebalance plugin rather than attempting to implement rebalancing yourself manually.

Yes, I will try it maybe today. However I only want to balance this one channel, so I have to use the other option that gives me more control instead of rebalance all channels.

[cdecker]

Yes, I will try it maybe today. However I only want to balance this one channel, so I have to use the other option that gives me more control instead of rebalance all channels.

You'll need to match up two channels to be rebalanced: one too much funds and another with too little, since rebalance will move funds from the over-full one to the empty one, otherwise where would the funds go to, or come from otherwise? This is what the rebalance plugin does.