search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.81k stars 887 forks source link

Core Lightning is incompatible with the Lightning Network - lacks support for payments #7180

Open nakoshi-satamoto opened 5 months ago

nakoshi-satamoto commented 5 months ago

I am not sure if it is just me or others also, but I have noticed that ever since updating to version 24 most payments fail.

I get alot of these messages, the htlc and node number varies though.

htlc 16 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)

then payment deadline expires.

I have even attempted to pay various invoices even with small payments, I also have some large channels (millions of sats) open with large and well connected nodes.

Posting this for visibility in case if others are experiencing this also. If there is something I can check to confirm if it is an issue with my node or corelightning I can do so.

nakoshi-satamoto commented 5 months ago

To clarify, it is uncertain if this is a problem wiht my node or something to do with core lightning.

vincenzopalazzo commented 5 months ago

What happens if you put disable-mmp inside your config file and restart your node?

nakoshi-satamoto commented 5 months ago

@vincenzopalazzo I was away from computer but I can try that now. Before I do that I want to know what that setting does. I am not finding any documentation on it. What is that setting?

vincenzopalazzo commented 5 months ago

the setting disable the multi path payment algorithm

nakoshi-satamoto commented 5 months ago

disable-mmp is not a recgonized command or config setting by my Core Lightning.

vincenzopalazzo commented 5 months ago

@nakoshi-satamoto is disable-mpp sorry

kilrau commented 4 months ago

@nakoshi-satamoto is disable-mpp sorry

Did this change anything for you? @nakoshi-satamoto

nakoshi-satamoto commented 4 months ago

@kilrau I actually did not need to.

Ever since I upgraded to core lightning verison 24.02 I kept getting alot of these errors in my logs for various channels. This also was an issue on V24.02.1

2024-04-15T04:27:36.026Z UNUSUAL lightningd: Bad gossip order: could not find channel 696011x284x1 for peer's channel update

After updating to the latest version of core lightning onto version 43.02.2 those gossip issues went away, transactions inbound and outbound are not failing anymore. Maybe it was related to gossip not working?

vincenzopalazzo commented 4 months ago

Maybe it was related to gossip not working?

probably yes

nakoshi-satamoto commented 4 months ago

No longer an issue

BitcoinMechanic commented 4 months ago

Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of Bad gossip order....)

nakoshi-satamoto commented 4 months ago

On 2024-05-05 18:52, BitcoinMechanic wrote:

Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of Bad gossip order....)

I am on 24.02.2 and I still get those bad gossip order messages. I am able to do payments though.

blapcity commented 3 months ago

also on 24.02.02 with high - payment failure rate and bad gossip order messages

crc32 commented 3 months ago

I'm also on 24.02.02 (using the BTCPay Docker image), and 100% payment failures unless paying direct channel partners, and tons of bad gossip order messages. Cannot query remote routes at all, even to targets that are direct partners of channel partners. Incoming payments and some through-routes seem to work though (the through routes may only be working to next-node termination though).

Example log entries:

2024-05-27T18:19:39.925Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat 2024-05-27T18:21:25.798Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat 2024-05-27T18:47:11.456Z INFO plugin-pay: cmd 34 partid 0: Payment fee constraint 2500msat is below exemption threshold, allowing a maximum fee of 5000msat

2024-05-27T18:21:25.803Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]} 2024-05-27T18:47:11.484Z INFO plugin-pay: cmd 34 partid 0: Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable. 2024-05-27T18:47:11.486Z INFO plugin-clnrest.py: RPC Error: {'code': 210, 'message': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 03b428ba4b48b524f1fa929203ddc2f0971c2077c2b89bb5b22fd83ed82ac2f7e1 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 500000msat}]}

2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 844051x1140x1 for peer's channel update 2024-05-27T18:47:55.749Z UNUSUAL lightningd: Bad gossip order: could not find channel 786203x525x0 for peer's channel update 2024-05-27T18:49:25.765Z UNUSUAL lightningd: Bad gossip order: could not find channel 746879x1261x2 for peer's channel update 2024-05-27T18:49:25.775Z UNUSUAL lightningd: Bad gossip order: could not find channel 820913x3568x0 for peer's channel update

UPDATE: I added disable-mpp to the startup file, with no effect.

nakoshi-satamoto commented 3 months ago

I said that I can still do payments, I am ammending my claim. I can receive inbound payments no problem but I do have a high rate of failure for outbound payments unless it is a direct peer. Ever since I upgraded to version 24 I have noticed this and it also coinciding with the gossip error messages. I delete gossip database and it still keeps getting gossip errors.

TTK-199837 commented 3 months ago

I am also experiencing this exact issue.

rustyrussell commented 3 months ago

DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.

Are you missing gossip? A healthy node should look like this:

$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062

That is, about 85000 public channels and 9000 nodes.

TTK-199837 commented 3 months ago

Well… in an attempt to make my core lightning node functional again I did delete the gossip store (but did save a backup! (This was… a week ago?)) should I just continue as is or cp gossip_store.backup to replace the “new” one? Have you not noticed any payment issues Rusty?

On Mon, Jun 3, 2024, at 9:26 PM, Rusty Russell wrote:

DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.

Are you missing gossip? A healthy node should look like this:

$ lightning-cli -H listchannels | grep -c ^short_channel_id 85921 $ lightning-cli -H listnodes | grep -c ^nodeid 9062 That is, about 85000 public channels and 9000 nodes.

— Reply to this email directly, view it on GitHub https://github.com/ElementsProject/lightning/issues/7180#issuecomment-2146579773, or unsubscribe https://github.com/notifications/unsubscribe-auth/AND3BCSLAHAFJBKAP3S2W4LZFU6XHAVCNFSM6AAAAABFLB774SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGU3TSNZXGM. You are receiving this because you commented.Message ID: @.***>

strnly commented 3 months ago

Found this issue and could relate instantly.

Same situation for me. BTCPay Server, CLN. Fat channels (7-12m sats) with major peers and payments to these peers fail 95%. Failed payments to Boltz, Strike, WoS despite having fat channels with them.

Incoming payments, no problem at all. Outgoing, 95%+ failure.

nakoshi-satamoto commented 3 months ago

On 2024-06-04 06:26, Rusty Russell wrote:

DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.

Are you missing gossip? A healthy node should look like this:

$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062

That is, about 85000 public channels and 9000 nodes.

What is wrong with deleting gossip? I have done that often times in version 24 to fix gossip issues. It re-populates fast. I deleted gossip yesterday after I upgraded to version 24.05 which claims to fix the gossip issues.

I just checked and I have 92910 channels, 16063 nodes.

blapcity commented 3 months ago

Did this resolve your issue with payments ?

nakoshi-satamoto commented 2 months ago

On 2024-06-07 02:33, blapcity wrote:

Did this resolve your issue with payments ?

I had the opportunity to test some payments today. I have 12 chans and almost 10,000,000 of outbound liquidity combined. Many of the peers I'm connected to are well connected also. I tried three random payments of less than 100,000 to different destinations and all of them failed. I still see gossip errors sometimes also, even after upgrading to 24.05 and rebuliding gossip. I'd say my gossip database is healthy also as I have 14427 nodes, 92599 chans in my view.

As of now, Core Lightning cannot send payments unless it is a directly connected peer. This is not functional.

nakoshi-satamoto commented 2 months ago

On 2024-06-10 19:46, Satoshi wrote:

I had the opportunity to test some payments today. I have 12 chans and almost 10,000,000 of outbound liquidity combined.

10 millions sats.

cdecker commented 2 months ago

Any chance we could get some more logs to try and hone in on the cause for the increased failure rate? From the above they all seem to come more or less directly after deleting gossip_store (fails immediately due to lack of network view, rather than taking some time and performing real attempts).

The gossip order is very likely to be unrelated, so ignore that maybe for a bit. It is mostly a warning that the peer has given us gossip messages in the incorrect order (update before announcement, or node before any of it's channels).

cdecker commented 2 months ago

One more thing, how many here are running btcpayserver? As it has been mentioned a number of times, I'd like to exclude that as a potential source.

compane commented 2 months ago

I have similar problems since upgrading to 24.05 without running btcpayserver. I'm only connected to boltz with a channel capacity of 1.000.000 sats. The logs below are from a failed transaction to phoenix wallet. Sending funds to wallet of satoshi was also not possible (same amount).

2024-06-15T21:25:26.486Z INFO    plugin-pay: cmd 34 partid 0: Initial limit on max HTLCs: 1305, Destination 027cd974e47086291bb8a5b0160a889c738f2712a703b8ea939985fd16f3aae67e has 87 channels, assuming 15 HTLCs per channel
2024-06-15T21:25:26.486Z INFO    lightningd: Sending 35000035msat in onion to deliver 35000000msat
2024-06-15T21:25:29.544Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 144 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:29.557Z INFO    plugin-pay: cmd 34 partid 0: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:29.589Z INFO    lightningd: Sending 35001105msat in onion to deliver 35000000msat
2024-06-15T21:25:32.612Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 145 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:32.623Z INFO    plugin-pay: cmd 34 partid 1: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:32.657Z INFO    lightningd: Sending 35001071msat in onion to deliver 35000000msat
2024-06-15T21:25:35.432Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 146 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:35.443Z INFO    plugin-pay: cmd 34 partid 2: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:35.481Z INFO    lightningd: Sending 35004070msat in onion to deliver 35000000msat
2024-06-15T21:25:38.787Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 147 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:38.798Z INFO    plugin-pay: cmd 34 partid 3: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:38.836Z INFO    lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:25:44.903Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 148 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:44.915Z INFO    plugin-pay: cmd 34 partid 4: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:44.955Z INFO    lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:25:51.109Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 149 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:51.120Z INFO    plugin-pay: cmd 34 partid 5: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:51.170Z INFO    lightningd: Sending 35003821msat in onion to deliver 35000000msat
2024-06-15T21:25:54.760Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 150 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:54.772Z INFO    plugin-pay: cmd 34 partid 6: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:54.820Z INFO    lightningd: Sending 35001040msat in onion to deliver 35000000msat
2024-06-15T21:25:57.924Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 151 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:25:57.933Z INFO    plugin-pay: cmd 34 partid 7: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:25:57.981Z INFO    lightningd: Sending 35001590msat in onion to deliver 35000000msat
2024-06-15T21:26:02.739Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 152 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:02.751Z INFO    plugin-pay: cmd 34 partid 8: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:02.797Z INFO    lightningd: Sending 35004170msat in onion to deliver 35000000msat
2024-06-15T21:26:05.668Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 153 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:05.679Z INFO    plugin-pay: cmd 34 partid 9: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:05.740Z INFO    lightningd: Sending 35003455msat in onion to deliver 35000000msat
2024-06-15T21:26:23.011Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 154 failed from 2th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:23.040Z INFO    plugin-pay: cmd 34 partid 10: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:23.131Z INFO    lightningd: Sending 18167140msat in onion to deliver 18167122msat
2024-06-15T21:26:23.153Z INFO    lightningd: Sending 16833926msat in onion to deliver 16832878msat
2024-06-15T21:26:29.574Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 155 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:29.585Z INFO    plugin-pay: cmd 34 partid 11: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:29.585Z INFO    plugin-pay: cmd 34 partid 11: Payment deadline expired, not retrying (partial-)payment 21229a3a6e8bb8d7001101d258bcf1f23ce2df2fc37406b25b1ad3a065193a4c/11
2024-06-15T21:26:30.010Z INFO    02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018-chan#28: htlc 156 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2024-06-15T21:26:30.021Z INFO    plugin-pay: cmd 34 partid 12: failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
2024-06-15T21:26:30.021Z INFO    plugin-pay: cmd 34 partid 12: Payment deadline expired, not retrying (partial-)payment 21229a3a6e8bb8d7001101d258bcf1f23ce2df2fc37406b25b1ad3a065193a4c/12
2024-06-15T21:26:30.022Z UNUSUAL plugin-clrest.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Ran out of routes to try after 13 attempts: see `paystatus`\",\"code\":210,\"attempts\":[{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":0,\"amount_msat\":35000000},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":1,\"amount_msat\":35000000,\"parent_partid\":0},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":2,\"amount_msat\":35000000,\"parent_partid\":1},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":3,\"amount_msat\":35000000,\"parent_partid\":2},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":4,\"amount_msat\":35000000,\"parent_partid\":3},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":5,\"amount_msat\":35000000,\"parent_partid\":4},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":6,\"amount_msat\":35000000,\"parent_partid\":5},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":7,\"amount_msat\":35000000,\"parent_partid\":6},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":8,\"amount_msat\":35000000,\"parent_partid\":7},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":9,\"amount_msat\":35000000,\"parent_partid\":8},{\"status\":\"pending\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":10,\"amount_msat\":35000000,\"parent_partid\":9},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":11,\"amount_msat\":18167122,\"parent_partid\":10},{\"status\":\"failed\",\"failreason\":\"failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)\",\"partid\":12,\"amount_msat\":16832878,\"parent_partid\":10}],\"fullType\":\"lightning\"}

2024-06-17T17:52:39.935Z UNUSUAL lightningd: Bad gossip order: could not find channel 812591x937x0 for peer's channel update

If this issue is related, I'm happy to share more information.

cdecker commented 2 months ago

Ok, I think I see what's happening. Just to be sure, can you provide the couple more lines above the snippet you posted? It contains the channel hints that we initialize based on our local view of the channels (listpeerchannels) which we then use to compute the actual route as an overlay on the gossip. It seems that we are picking a local channel that is either not fully reestablished or does not have sufficient balance. Then we somehow track the limits wrong and retry the same channel over and over again. At least skimming the logs, that's my best theory yet. More data and logs will help us pin this down and fix it.

Thanks for the logs, they are incredibly helpful 👍

cdecker commented 2 months ago

Also, please set your log levels to debug.

compane commented 2 months ago

Thank your for taking the time. I set log levels to debug and did three transactions.

1 sat --> succeeded 10 sats --> succeeded 1000 sats --> failed I had a 128000 sats transaction in the past which succeeded as well.

cln_debug_failed_1000sats_primal.txt cln_debug_succeeded_10sats_phoenix.txt cln_debug_succeeded_1sat_primal.txt

blapcity commented 2 months ago

My friend runs an LND node and I'm his only channel. I have 25 large channels and I cannot send a payment to Strike but he can with 100% success make payments to strike despite his only channel being with me. Not sure if this is helpful but it confirms that there is an issue.

lightningcheckout commented 2 months ago

I recognize this issue as well. When local balance in direct channel isn't enough payments are failing most of the times. @cdecker is this the part you are looking for:

Multiple times it has occured that local balance in the WoS channel is not sufficient, but in the Kappa channel (which is also WoS) is. But in such cases payments to WoS invoices are failing. 

2024-06-22T07:10:13.415Z DEBUG   plugin-pay: cmd 34 partid 0: Received getchaininfo blockcount=848999, headercount=848999
2024-06-22T07:10:13.416Z DEBUG   plugin-pay: cmd 34 partid 0: waitblockheight reports syncheight=848999
2024-06-22T07:10:13.425Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 796747x1740x0/1: enabled true, estimated capacity 5195608458msat
2024-06-22T07:10:13.426Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 829208x14004966x61721/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.426Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 5960895x10194222x36605/1: enabled false, estimated capacity 4294967295msat
2024-06-22T07:10:13.426Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 848103x2170x1/0: enabled true, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 10454817x11377454x13607/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 819140x2281x1/1: enabled true, estimated capacity 54359915msat
2024-06-22T07:10:13.427Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 5591485x9388989x13342/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.427Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 826396x2260x0/1: enabled true, estimated capacity 1248975146msat
2024-06-22T07:10:13.428Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 4282556x1477417x5590/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.428Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 847889x1670x0/0: enabled true, estimated capacity 1581645457msat
2024-06-22T07:10:13.428Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 14226214x12490668x34238/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.428Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 831632x831x8/1: enabled true, estimated capacity 197645333msat
2024-06-22T07:10:13.429Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 12419924x6069763x832/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.429Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 830097x1343x0/0: enabled true, estimated capacity 3583467msat
2024-06-22T07:10:13.429Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 5448693x1437927x55335/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.429Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 794802x1236x1/1: enabled true, estimated capacity 56440577msat
2024-06-22T07:10:13.430Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 15232671x14304892x37732/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.437Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 836553x1098x0/1: enabled true, estimated capacity 12981990msat
2024-06-22T07:10:13.438Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 15804156x10261040x52402/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.438Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 825037x421x0/1: enabled true, estimated capacity 6291173857msat
2024-06-22T07:10:13.438Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 4909945x337710x31253/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.438Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 806422x1945x1/1: enabled true, estimated capacity 8341msat
2024-06-22T07:10:13.439Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 5015597x10726383x19215/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 846557x3093x0/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 1880118x5887875x11970/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.439Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 826544x3884x1/1: enabled true, estimated capacity 4673825msat
2024-06-22T07:10:13.440Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 15140828x5950323x21109/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.440Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 774864x1716x1/1: enabled true, estimated capacity 250150121msat
2024-06-22T07:10:13.440Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 8852482x10869402x60338/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.440Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 776826x3067x1/1: enabled true, estimated capacity 10719488msat
2024-06-22T07:10:13.441Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 8061376x5028423x11465/1: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.441Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 844759x290x1/0: enabled true, estimated capacity 59431000msat
2024-06-22T07:10:13.441Z DEBUG   plugin-pay: cmd 34 partid 0: Added a channel hint for 4959791x10771686x64007/0: enabled false, estimated capacity 0msat
2024-06-22T07:10:13.552Z DEBUG   plugin-pay: cmd 34 partid 0: After filtering routehints we're left with 0 usable hints
2024-06-22T07:10:13.553Z DEBUG   plugin-pay: cmd 34 partid 0: Not using a routehint
2024-06-22T07:10:13.553Z DEBUG   plugin-pay: cmd 34 partid 0: The destination is directly reachable including attempts without routehints
2024-06-22T07:10:13.650Z INFO    plugin-pay: cmd 34 partid 0: Initial limit on max HTLCs: 23475, Destination 035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226 has 1565 channels, assuming 15 HTLCs per channel
2024-06-22T07:10:13.650Z DEBUG   plugin-pay: cmd 34 partid 0: Created outgoing onion for route: 774864x1716x1 -> 841217x2174x0 -> 841809x2912x2 -> 830185x454x0
2024-06-22T07:10:13.653Z INFO    lightningd: Sending 241409341msat in onion to deliver 241408000msat
2024-06-22T07:10:22.444Z INFO    033d38e07e2541628214c8184cf6c562311059872363e9f538b2f45434418b5bc5-chan#4: htlc 14690 failed from 2th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
cdecker commented 2 months ago

@lightningcheckout Im assuming that there are more HTLCs failing? Individual HTLCs failing is to be expected, as we don't have the balances is the open network and so LN just tries, learns then retries new HTLCs. So one htlc going awry is not suspicious, but if we end up retrying the same channel over and over again, then that'd be something to look into.

lightningcheckout commented 2 months ago

Indeed, payment is failing at the timeout. Can send you the complete log later today if needed?

cdecker commented 2 months ago

That would be very helpful indeed.

nakoshi-satamoto commented 2 months ago

@cdecker

I was travelling and am back now. I captured a state of all my channels and their balances, I set my log level to debug and attempted two seperate payments to two different destinations. All failed. I would consider this issue to be of the highest priority for core lightning, because at the moment Core Lightning simply is unable to do payments unless if it is to a directly connected peer. I am unable to use my node for payments because of this. I am only keeping it running in the meantime to paitently await in hope that one day this issue may be resolved.

Attached is the state of my router and channels and balances, then the debug log, then the two payment errors that went to the standard output. I am not using BTCPay, I am using Core Lightning in command line. I am using the pay command. (Does the debug log contain sensitive info like private keys?)

Thank you for your work to you and the rest of the devs.

LN-state-2024-Jun-22.txt ln-log.txt 7778sat-payment-failed.txt 5000sat-payment-failed.txt

cdecker commented 2 months ago

Thanks for the logs @nakoshi-satamoto :+1: We are very aware that some users are having a bad time performing payments with CLN, and it is our top priority. Please be aware that debugging a complex piece of software is not straightforward, and the lack of progress is not due to us not trying, but rather, there was very little actionable information to debug the issue. With the provided logs I hope we can narrow down the issue to the root cause, and hopefully we'll find a fix soon.

cdecker commented 2 months ago

I went and looked at the 5000sat payment, that @nakoshi-satamoto added logs for, and I reconstructed the following timeline:

Timeline for the 5000sat payment:

The payment ends up failing after the 60 second send timeout expires. It tried a whole bunch of different paths, but never succeeded, because all paths reported insufficient balances either in the 1- or 2-hop neighborhood. As frustrating as it may seem, this is how the LN protocol works, and the failure rate is due to network weather, not the behavior of the sender.

We are working on remembering channel statuses (as in the channel hints being added dynamically) across payments, which may reduce the attempts required to find a working path, in case of repeated payments, but that does not help nodes that don't perform many payments, thus have a limited view of the network weather.

As the issues arise in the <2-hop neightborhood, I'd strongly suggest opening new channels to peers that are better connected, and that take better care of balancing their channels. It really looks like you got very unlucky.

The following peers or peers-of-peers caused your payment to fail:

Reporting Node ID Channel ID Name HTLC
026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 824675x381x0 Boltz 3922
03c82c83da9e495a5f72fcfe1671f3ed84452234b82ef6f2403f24d08d59aa8295 847399x918x0 Nerk 3923
026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 809661x1625x0 Boltz 3924
039174f846626c6053ba80f5443d0db33da384f1dde135bf7080ba1eec465019c3 832721x2744x2 - 3925
039174f846626c6053ba80f5443d0db33da384f1dde135bf7080ba1eec465019c3 847700x1843x0 - 3926
02a98e8c590a1b5602049d6b21d8f4c8861970aa310762f42eae1b2be88372e924 830185x454x0 - 3927
026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 841648x644x0 Boltz 3928
026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2 769934x630x0 Boltz 3929

It's too bad that those no-name nodes didn't have sufficient capacity, since they were the last hop before the destination WoS, but that's life.

So from what I can tell, it tried really hard to deliver the payment. Some slow nodes along the routes caused the payment to only ever try 8 paths, which sadly ended up with insufficient balance errors.

cdecker commented 2 months ago

So there are a number of initiatives we are looking into right now, and some that could use some operator input to fine-tune them:

We are of course open to suggestions.

cdecker commented 2 months ago

And the second payment has the following routes (with the failing channel with insufficient balance boldened):

Notice you always pick Boltz as first hop, maybe excluding that peer from being used could change success rate? It seems like Boltz is so attractive to routing that the randomization we add never ends up picking another peer to go through, but then it runs into balance issues.

cdecker commented 2 months ago

@lightningcheckout looking into your logs, it also appears like your node ends up failing all attempts due to insufficient balance in the 1-hop and 2-hop neighborhoods.

nakoshi-satamoto commented 2 months ago

Thank you very much @cdecker . You are very much appreciated.

So as I understand, it is just that my router is in a bad state for outbound liquidity, even though I have a large channel open with Boltz.

I am going to test some payments with excluding Boltz chan I have open. I cannot figure out how to exclude a channel when making payments, I did read the manuals. Does anyone know how to exclude a channel for payments?

Then I was going to get more outbound liquidity on my side, maybe by rebalancing come chans and do testing again.

nakoshi-satamoto commented 2 months ago

If I cannot figure out how to exclude a chan for outbound payments, I may just close my boltz channel then use that to try to rebalance some other chans.

lightningcheckout commented 2 months ago

Did you check: https://docs.corelightning.org/reference/lightning-pay ? The exclude syntax is given here. I also use sometime the paythrough plugin to tell CLN which channel to use for the outgoing payment.

nakoshi-satamoto commented 2 months ago

On 2024-06-23 18:45, lightningcheckout wrote:

Did you check: https://docs.corelightning.org/reference/lightning-pay ? The exclude syntax is given here. I also use sometime the paythrough plugin to tell CLN which channel to use for the outgoing payment.

That is the first manual page I checked. That documentation is lacking the information I need to craft a working command. It returns an error asking for amount in msats, (shouldn't this already be included in the bolt11 invoice?). I am not able to figure out the proper syntax even when I do manually input the amount again.

So unless anyone has knowledge on how to use the exclude flag in the pay command, this is not a valid option for me. So my only option is to install the plugin to pay through a certain channel. I will try that later on as a test to see if I can pay through one of my other channels that is not Boltz. Then I shall rebalance a chan or two to get more outbound liquidity on my side and try payments again.

daywalker90 commented 2 months ago

Assuming there really is an amount in the invoice you use the command like this:

lightning-cli pay -k bolt11=<invoice> exclude='["564334x877x1/0", "564334x877x1/1"]'
nakoshi-satamoto commented 2 months ago

On 2024-06-24 00:31, daywalker90 wrote:

Assuming there really is an amount in the invoice you use the command like this:

lightning-cli pay -k bolt11=<invoice> exclude='["564334x877x1/0",
"564334x877x1/1"]'

Thank you. When excluding the Boltz chan, I did a test of three payments to three different destinations, small payments. Two of the three still failed. I closed my boltz chan and rebalanced to bring back more outbound liquidity across different channels. Now payments are successful.

So it seems that although I had high amount of outbound liquidity on my Boltz chan, and some outbound liquidity on other chans, perhaps my node was just in a bad position in terms of liquidity on the network to reach destinations.

compane commented 2 months ago

@nakoshi-satamoto Was it the LND or CLN Boltz node you were connected to?

nakoshi-satamoto commented 2 months ago

On 2024-06-24 09:35, compane wrote:

@nakoshi-satamoto Was it the LND or CLN Boltz node you were connected to?

pubkey 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2

So this is Boltz, not Boltz|CLN. I am using CLN

nakoshi-satamoto commented 2 months ago

On 2024-06-24 15:51, Satoshi wrote:

On 2024-06-24 09:35, compane wrote:

@nakoshi-satamoto Was it the LND or CLN Boltz node you were connected to?

pubkey 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2

So this is Boltz, not Boltz|CLN. I am using CLN

I mean, i am using CLN as in Core Lightning as my router. I was not connected to Boltz|CLN

compane commented 2 months ago

So I‘m connected to the Boltz CLN node only. I asked the Boltz guys if they are aware of any routing issues. They denied that. Am 24.06.2024 um 15:54 schrieb antifiat, cypherpunk, destroyer of central banks @.***>: On 2024-06-24 15:51, Satoshi wrote:

On 2024-06-24 09:35, compane wrote:

@nakoshi-satamoto Was it the LND or CLN Boltz node you were connected to?

pubkey 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2

So this is Boltz, not Boltz|CLN. I am using CLN

I mean, i am using CLN as in Core Lightning as my router. I was not connected to Boltz|CLN

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>