search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.64k stars 2.07k forks source link

[bug]: FAILURE_REASON_NO_ROUTE to a local channel - not fee related #8639

Open charltonh opened 5 months ago

charltonh commented 5 months ago

Background

Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE To a local channel that can receive payments just fine.

This used to always work until the update from LND-0.15.3-beta to LND-0.17.4-beta

Your environment

Steps to reproduce

Cannot pay an invoice generated by BLW wallet. Used to be able to.

Expected behaviour

Payment should go through

Actual behaviour

bach /home/charlton [36]>lncli payinvoice lnbc100n1pnpd0exrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5d3pua7d80tt6e55a5xq8guhw7k8q6qyurghu0wuxv6lzjsh2f0essp5j72py53mycs6u5sndnf3vxma67ra37nyrtdw9dzjw4h2hpw5fhcqxqyz5vq9qypqsq9mjnse9d94ja7d49d47q3y0d9l5zv30qn5yy59j3m9ztakzfanfrwp4s2kkm2x7l6mv5gdzmeffv84exd5kxkqzalpravr3ufa28qmcqwmd5m3
Payment hash: 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3
Description:
Amount (in satoshis): 10
Fee limit (in satoshis): 10
Destination: 03aa0688276455fdb4c01e4615d4147a816cbbf80c0474e681e3584b95b653a7ca
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

-- LND log:

2024-04-10 11:04:37.589 [WRN] CRTR: ShortChannelID=647847:920:0: link not found: channel link not found
2024-04-10 11:04:37.590 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-10 11:04:37.590 [WRN] CRTR: Failed to find route for payment 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3: unable to find a path to destination
2024-04-10 11:04:37.607 [ERR] CRTR: Payment 6c43cef9a77ad7acd29da1807472eef58e0d009c1a2fc7bb8666be2942ea4bf3 failed: no_route

-- LND log after receiving a txn, then immediately trying to send a txn to the same peer:

2024-04-10 11:12:17.783 [INF] CRTR: Pruning channel graph using block 000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d (height=838633)
2024-04-10 11:12:17.797 [INF] CHDB: Pruned unconnected node 02d9cd98233e1535c9270c0b541cf02e68ce01e5fc4a9ff0d4af17051fff7f3ac3 from channel graph
2024-04-10 11:12:17.797 [INF] CHDB: Pruned 1 unconnected nodes from the channel graph
2024-04-10 11:12:17.805 [INF] CRTR: Block 000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d (height=838633) closed 2 channels
2024-04-10 11:12:17.854 [INF] NTFN: New block: height=838633, sha=000000000000000000021f66080f8a9758b2cf171cd4f859dadd9dfbe53ae03d
2024-04-10 11:12:17.854 [INF] UTXN: Attempting to graduate height=838633: num_kids=0, num_babies=0
2024-04-10 11:12:45.032 [INF] CRTR: Processed channels=0 updates=125 nodes=32 in last 1m0.000112615s
2024-04-10 11:12:46.896 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-10 11:12:46.897 [WRN] CRTR: Failed to find route for payment 810d0ab626ab1477367a06e359be56332bfd30caba4c76082888f10f07f8f8d3: unable to find a path to destination
2024-04-10 11:12:46.914 [ERR] CRTR: Payment 810d0ab626ab1477367a06e359be56332bfd30caba4c76082888f10f07f8f8d3 failed: no_route

-- Additional info:

bach /home/charlton [30]>lncli getnodeinfo 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac
{
    "node": {
        "last_update": 0,
        "pub_key": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
        "alias": "",
        "addresses": [],
        "color": "#000000",
        "features": {},
        "custom_records": {}
    },
    "num_channels": 1,
    "total_capacity": "8116263",
    "channels": []
}
ziggie1984 commented 5 months ago

Can you post the channel-constraints and policy of the channel you think should send/route the payment?

charltonh commented 5 months ago 
bach /home/charlton [63]>lncli --help | grep -i constraint
bach /home/charlton [64]>lncli --help | grep -i policy
     updatechanpolicy   Update the channel policy for all channels, or a single channel.
bach /home/charlton [65]>lncli listchannels
{
    "channels": [
        {
            "active": true,
            "remote_pubkey": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
            "channel_point": "233d068b94254376bf95fee1cd126baeef28bdc423ada1c59bef13541b7b128c:0",
            "chan_id": "712315309580091392",
            "capacity": "8116263",
            "local_balance": "7999430",
            "remote_balance": "115041",
            "commit_fee": "1792",
            "commit_weight": "724",
            "fee_per_kw": "2474",
            "unsettled_balance": "0",
            "total_satoshis_sent": "490110453",
            "total_satoshis_received": "498109884",
            "num_updates": "4637",
            "pending_htlcs": [],
            "csv_delay": 2016,
            "private": true,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "40581",
            "remote_chan_reserve_sat": "81162",
            "static_remote_key": false,
            "commitment_type": "LEGACY",
            "lifetime": "68794",
            "uptime": "92",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "40581",
                "dust_limit_sat": "573",
                "max_pending_amt_msat": "16777215000",
                "min_htlc_msat": "10000",
                "max_accepted_htlcs": 25
            },
            "remote_constraints": {
                "csv_delay": 975,
                "chan_reserve_sat": "81162",
                "dust_limit_sat": "546",
                "max_pending_amt_msat": "8035101000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0",
            "peer_alias": "unable to lookup peer alias: alias for node not found",
            "peer_scid_alias": "0",
            "memo": ""
        },
{...snip...}

This is the relevant channel. I'm not sure why in the payment the destination is not the same address.

Here's what RTL shows me about the channel: oob_capt3

charltonh commented 5 months ago

Again, this channel worked with v0.15.3 Even if this turns out to be constraint/policy/fee related, the lack of that info in the error msg to a local channel is a bug.

But I suspect that rather, this is a bug with the overlooking of some sort of backwards compatibility.

ziggie1984 commented 5 months ago

So you want to pay a local connected node ?

So you wanna pay pub_key: 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac ?

Because something is strange in the invoice you are providing it has the following route hints:

    "route_hints":  [
        {
            "hop_hints":  [
                {
                    "node_id":  "03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe",
                    "chan_id":  "712315309580091392",
                    "fee_base_msat":  1000,
                    "fee_proportional_millionths":  1,
                    "cltv_expiry_delta":  40
                }
            ]
        },
        {
            "hop_hints":  [
                {
                    "node_id":  "03144fcc73cea41a002b2865f98190ab90e4ff58a2ce24d3870f5079081e42922d",
                    "chan_id":  "193815628646246820",
                    "fee_base_msat":  1000,
                    "fee_proportional_millionths":  100,
                    "cltv_expiry_delta":  144
                }
            ]
        }
    ],

probably node_key 03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe cannot be found, try creating an invoice without route_hints if its a direct channel

charltonh commented 5 months ago

03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe is the node the payment is coming from running LND.

ziggie1984 commented 5 months ago

Can you set the debug-level of the CRTR subsystem to debug and try the payment again, and show the corresponding logs, thank you.

So I translated the short-chan-id: 712315309580091392=>647847:920:0

So the channel link seems to be offline when sending:

2024-04-10 11:04:37.589 [WRN] CRTR: ShortChannelID=647847:920:0: link not found: channel link not found

can you verify that during the time of sending your channel is active and online ?

charltonh commented 5 months ago

Looks like you may be on to something.

bach /home/charlton [47]>lncli payinvoice lnbc10m1pnpegxjrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5pyktg56dh2afq0395h54ljpegq0l2g77g8nm58kdt4y8j0swlxrqsp5urfj6qv42pue5ptttfgeuhmpj94gylsahehw7l2dztaaq6rl7e2qxqyz5vq9qypqsqc2aec72hszja3v77xaq6adzqz7573c0k442kekw2hmhxygshlk8krf055a2avevtl6u4vpua5wsqmruafvplkw78swv9phfakka9u2gp9uytpu
Payment hash: 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Description:
Amount (in satoshis): 1000000
Fee limit (in satoshis): 50000
Destination: 020ca2d63385f8ebbce3e2d5e85ae9a63d16c1737e5248f70bf8306f62e8d279c8
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
bach /home/charlton [48]>

lnd server log:

2024-04-15 12:51:13.719 [DBG] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 in state terminate=false, active_shards=0, rem_value=1000000000 mSAT, fee_limit=50000000 mSAT
2024-04-15 12:51:13.719 [DBG] CRTR: PaymentSession(092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986): pathfinding for amt=1000000000 mSAT
2024-04-15 12:51:13.719 [WRN] CRTR: ShortChannelID=685493:409:1: link not found: channel link not found
2024-04-15 12:51:13.719 [DBG] CRTR: Pathfinding absolute attempt cost: 1100 sats
2024-04-15 12:51:13.719 [DBG] CRTR: Pathfinding perf metrics: nodes=2, edges=1, time=189.112µs
2024-04-15 12:51:13.719 [DBG] CRTR: PaymentSession(092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986): not splitting because destination doesn't declare MPP or AMP
2024-04-15 12:51:13.719 [WRN] CRTR: Failed to find route for payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986: unable to find a path to destination
2024-04-15 12:51:13.719 [DBG] CRTR: Marking payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 permanently failed with no route: no_route
2024-04-15 12:51:13.736 [DBG] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 in state terminate=true, active_shards=0, rem_value=1000000000 mSAT, fee_limit=50000000 mSAT
2024-04-15 12:51:13.736 [ERR] CRTR: Payment 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986 failed: no_route

The channel is not offline though. I can pay from the channel to the lnd node.

charltonh commented 5 months ago

When I do a lncli listpeers, here is the relevant channel:

        {
            "pub_key": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
            "address": "10.8.0.1:56554",
            "bytes_sent": "1283",
            "bytes_recv": "549",
            "sat_sent": "490110453",
            "sat_recv": "498123700",
            "inbound": true,
            "ping_time": "281965",
            "sync_type": "PASSIVE_SYNC",
            "features": {
                "1": {
                    "name": "data-loss-protect",
                    "is_required": false,
                    "is_known": true
                },
                "3": {
                    "name": "initial-routing-sync",
                    "is_required": false,
                    "is_known": true
                },
                "7": {
                    "name": "gossip-queries",
                    "is_required": false,
                    "is_known": true
                },
                "9": {
                    "name": "tlv-onion",
                    "is_required": false,
                    "is_known": true
                }
            },
            "errors": [],
            "flap_count": 3601,
            "last_flap_ns": "1713207047937542137",
            "last_ping_payload": "67"
        }

Not sure if this is related, but it seems to have a lot fewer features listed than other channels.

ziggie1984 commented 5 months ago

Can you look for this INFO log:

Adding live link chan_id=XXX, short_chan_id=685493:409:1 this adds the channel to the switch, do you see those logs for your other channels with different chan_ids obviously ?

ziggie1984 commented 5 months ago

can you maybe set every subsystem to debug, disconnect to the peer, reconnect and send the payment and post the logs, that would be super helpful 🙏

charltonh commented 5 months ago

Ok attaching some logs. I cannot isolate which logs are relevant but I cut-and-paste the portions while doing what you asked.

debug.txt

ziggie1984 commented 5 months ago

seems like one channel is active: Adding live link chan_id=8c127b1b5413ef9bc5a1ad23c4bd28efae6b12cde1fe95bf764325948b063d23, short_chan_id=647847:920:0 to peer 025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac

ziggie1984 commented 5 months ago

Ok I think I found the problem, will report soon.

ziggie1984 commented 5 months ago

whats the following output on your sending node:

lncli getchaninfo 712315309580091392 because I think you don't have this channel in your graph, but its a private direct channel which means you need to have it locally.

ziggie1984 commented 5 months ago

can you try an update of the channel policy and see whether this injects the channel back into the graph (updatechanpolicy)

charltonh commented 5 months ago

lncli getchaninfo 712315309580091392

bach /home/charlton [65]>lncli getchaninfo 712315309580091392
{
"channel_id": "712315309580091392",
"chan_point": "233d068b94254376bf95fee1cd126baeef28bdc423ada1c59bef13541b7b128c:0",
"last_update": 1712765355,
"node1_pub": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
"node2_pub": "03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe",
"capacity": "8116263",
"node1_policy": null,
"node2_policy": {
"time_lock_delta": 40,
"min_htlc": "10000",
"fee_base_msat": "1000",
"fee_rate_milli_msat": "1",
"disabled": false,
"max_htlc_msat": "8116263000",
"last_update": 1712765355,
"custom_records": {}
},
"custom_records": {}
}
charltonh commented 5 months ago

After updating the channel policy from 1000 msat to 1100, I get:

bach /home/charlton [71]>lncli getchaninfo 712315309580091392
{
    "channel_id": "712315309580091392",
    "chan_point": "233d068b94254376bf95fee1cd126baeef28bdc423ada1c59bef13541b7b128c:0",
    "last_update": 1713218624,
    "node1_pub": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
    "node2_pub": "03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe",
    "capacity": "8116263",
    "node1_policy": null,
    "node2_policy": {
        "time_lock_delta": 40,
        "min_htlc": "10000",
        "fee_base_msat": "712315309580091392",
        "fee_rate_milli_msat": "2",
        "disabled": false,
        "max_htlc_msat": "8116263000",
        "last_update": 1713218624,
        "custom_records": {}
    },
    "custom_records": {}
}
bach /home/charlton [72]>

Still unable to make a payment to it:

bach /home/charlton [72]>lncli payinvoice lnbc10m1pnpegxjrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5pyktg56dh2afq0395h54ljpegq0l2g77g8nm58kdt4y8j0swlxrqsp5urfj6qv42pue5ptttfgeuhmpj94gylsahehw7l2dztaaq6rl7e2qxqyz5vq9qypqsqc2aec72hszja3v77xaq6adzqz7573c0k442kekw2hmhxygshlk8krf055a2avevtl6u4vpua5wsqmruafvplkw78swv9phfakka9u2gp9uytpu
Payment hash: 092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Description:
Amount (in satoshis): 1000000
Fee limit (in satoshis): 50000
Destination: 020ca2d63385f8ebbce3e2d5e85ae9a63d16c1737e5248f70bf8306f62e8d279c8
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   092cb4534dbaba903e25a5e95fc839401ff523de41e7ba1ecd5d48793e0ef986
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
bach /home/charlton [73]>
ziggie1984 commented 5 months ago

Ok so the problem is the following:

your node as no features according from the point of view of the sender:

{
    "node": {
        "last_update": 0,
        "pub_key": "025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac",
        "alias": "",
        "addresses": [],
        "color": "#000000",
        "features": {},
        "custom_records": {}
    },
    "num_channels": 1,
    "total_capacity": "8116263",
    "channels": []
}

the path-finding will not allow a node which as no features set:

https://github.com/lightningnetwork/lnd/blob/07b6af41dbe2a5a1c85e5c46cc41019b64640d90/routing/pathfind.go#L1047-L1050

so I will think of a potential fix, and also increase the logs so that one knows immediately where the problem is when this occurs again. We know the features for a direct node without having them in the graph.

charltonh commented 5 months ago

I noticed the new release did not fix the problem. Is there some way I can "give" features to this existing channel?

Not sure how it's always worked before unless this is new non-backwards compatible functionality or the feature names changed. I'd close and re-open the channel if fees weren't astronomical right now.

ziggie1984 commented 5 months ago

For the records, both nodes run LND 17.5 now ?

charltonh commented 5 months ago

For the records, both nodes run LND 17.5 now ?

No the channel I'm trying to fill is my BLW lightning wallet on Android.

ziggie1984 commented 5 months ago

Working on a patch, will release by the end of tomorrow.

ziggie1984 commented 4 months ago

Ok looked again into your case and something stand out:

Your invoices have always a new destination:

For example: lnbc10m1pnpegxjrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5pyktg56dh2afq0395h54ljpegq0l2g77g8nm58kdt4y8j0swlxrqsp5urfj6qv42pue5ptttfgeuhmpj94gylsahehw7l2dztaaq6rl7e2qxqyz5vq9qypqsqc2aec72hszja3v77xaq6adzqz7573c0k442kekw2hmhxygshlk8krf055a2avevtl6u4vpua5wsqmruafvplkw78swv9phfakka9u2gp9uytpu has 020ca2d63385f8ebbce3e2d5e85ae9a63d16c1737e5248f70bf8306f62e8d279c8

your first invoice lnbc100n1pnpd0exrzjq0y5sc34k3lz05tj043appm504d92xu0wh2837w6u77veaa3ckk0uz0z5uqq8xqqqqqqqqlgqqqqqqgq9qrzjqv2ylnrne6jp5qpt9pjlnqvs4wgwfl6c5t8zf5u8pag8jzq7g2fz6q4sjfyezc8f5sqqqqlgqqqqqeqqjqdqqcqzgapp5d3pua7d80tt6e55a5xq8guhw7k8q6qyurghu0wuxv6lzjsh2f0essp5j72py53mycs6u5sndnf3vxma67ra37nyrtdw9dzjw4h2hpw5fhcqxqyz5vq9qypqsq9mjnse9d94ja7d49d47q3y0d9l5zv30qn5yy59j3m9ztakzfanfrwp4s2kkm2x7l6mv5gdzmeffv84exd5kxkqzalpravr3ufa28qmcqwmd5m3 destination: 03aa0688276455fdb4c01e4615d4147a816cbbf80c0474e681e3584b95b653a7ca

and the channel 712315309580091392 connects:

025fdd3cf396297c9cc8d9413927f82a4388ffa14fba94c4937337f41bdbc5b5ac and 03c9486235b47e27d1727d63d087747d5a551b8f75d478f9dae7bcccf7b1c5acfe

could you set all the debug settings to trace and retry the payment and post the log files. I still need a bit more details in your case