High failure rate for payment forwards

gallizoltan commented 1 year ago

Issue and Steps to Reproduce

My node has a very high failure rate for payment forwards. It is about 1 successful forward out of 100 attempts. What I tried:

I checked the channels to have enough liquidity for these transactions.
My internet connection is good.
I removed all the plugins, just for sure.

Can anyone tell me why this is happening?

lightning-cli listforwards
{
   "forwards": [
   .......
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20767,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7675,
         "in_msatoshi": 97207499,
         "in_msat": "97207499msat",
         "out_msatoshi": 97207499,
         "out_msat": "97207499msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671220948.847,
         "resolved_time": 1671220958.826
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20768,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7678,
         "in_msatoshi": 56499761,
         "in_msat": "56499761msat",
         "out_msatoshi": 56499761,
         "out_msat": "56499761msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671220988.069,
         "resolved_time": 1671220991.110
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20769,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7679,
         "in_msatoshi": 56498705,
         "in_msat": "56498705msat",
         "out_msatoshi": 56498705,
         "out_msat": "56498705msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221016.320,
         "resolved_time": 1671221019.977
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20770,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7680,
         "in_msatoshi": 56499987,
         "in_msat": "56499987msat",
         "out_msatoshi": 56499987,
         "out_msat": "56499987msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221023.580,
         "resolved_time": 1671221027.476
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20771,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7684,
         "in_msatoshi": 97112807,
         "in_msat": "97112807msat",
         "out_msatoshi": 97112807,
         "out_msat": "97112807msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221075.921,
         "resolved_time": 1671221084.563
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20772,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7685,
         "in_msatoshi": 97512194,
         "in_msat": "97512194msat",
         "out_msatoshi": 97512194,
         "out_msat": "97512194msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221076.764,
         "resolved_time": 1671221088.066
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20773,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7686,
         "in_msatoshi": 97123194,
         "in_msat": "97123194msat",
         "out_msatoshi": 97123194,
         "out_msat": "97123194msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221079.736,
         "resolved_time": 1671221090.994
      },
      {
         "in_channel": "744677x2145x1",
         "in_htlc_id": 20774,
         "out_channel": "767492x1627x1",
         "out_htlc_id": 7691,
         "in_msatoshi": 56499761,
         "in_msat": "56499761msat",
         "out_msatoshi": 56499761,
         "out_msat": "56499761msat",
         "fee": 0,
         "fee_msat": "0msat",
         "status": "failed",
         "style": "tlv",
         "received_time": 1671221128.496,
         "resolved_time": 1671221137.684
      }
   ]
}

`getinfo` output

{
   "id": "03d67f36c4f81789e2fe425028bacc96b199813eae426c517f589a45f1136c1fe5",
   "alias": "Jubilee",
   "color": "dc42f4",
   "num_peers": 43,
   "num_pending_channels": 0,
   "num_active_channels": 40,
   "num_inactive_channels": 2,
   "address": [
      {
         "type": "ipv4",
         "address": "80.99.6.12",
         "port": 9735
      },
      {
         "type": "torv3",
         "address": "ihhsjdncang7ve2rvhuunvy45bxvmh2qwz3vh7mohbouizd36ywnxeid.onion",
         "port": 9735
      }
   ],
   "binding": [
      {
         "type": "ipv6",
         "address": "::",
         "port": 9735
      },
      {
         "type": "ipv4",
         "address": "0.0.0.0",
         "port": 9735
      }
   ],
   "version": "22.11.1",
   "blockheight": 767702,
   "network": "bitcoin",
   "msatoshi_fees_collected": 686547360,
   "fees_collected_msat": "686547360msat",
   "lightning-dir": "/home/user/.lightning/bitcoin",
   "our_features": {
      "init": "08a000080269a2",
      "node": "88a000080269a2",
      "channel": "",
      "invoice": "02000000024100"
   }
}

vincenzopalazzo commented 1 year ago

Yep! your data are true, this is a data collection with lnmetrics https://bruce.lnmetrics.info/metrics that give also some information on when we fail and where we fail

whitslack commented 1 year ago

I don't think it's that uncommon.

lightning=> select sum(out_payments_offered), sum(out_payments_fulfilled) from channels;
   sum    |   sum   
----------+---------
 45286579 | 1133195
(1 row)

= My fulfillment rate is about 2.5%.

cvx35isl commented 1 year ago

my fulfillment rate is (334 / 105761.0) *100 = 0.31%

i tried improving this with the feeadjuster plugin, but no change

of which are very few failures where my node is the first or last hop and it shows some actual error. Thats payments to scid=666666x6666x6 and similar, or where my peers are unreachable.

port monitoring from outside shows its not node connectivity

so I am lost as to what I can improve

Any thoughts on why there is so much trying going on in the network?

vincenzopalazzo commented 1 year ago

i tried improving this with the feeadjuster plugin, but no change

This is not the solution, There is an apparent problem in pathfinding, and people are working. I am sorry for the bad news.

It is not just your node that have bad forwards rating https://lnmetrics.info/analysis

Ah more important, if you are forwarding this means that you are not choosing the path, so you can not do anything about it if the sender choose the wrong path

cvx35isl commented 1 year ago

interesting; from the Mastering the Lightning Network book my understanding was that pathfinding is something each node does "offline" based on his collected knowledge about the network ... no need for dumb trying. Therefore I never expected such a high failure rate.

Is there any public discussion somewhere about whats wrong with pathfinding?

vincenzopalazzo commented 1 year ago

interesting; from the Mastering the Lightning Network book my understanding was that pathfinding is something each node does "offline" based on his collected knowledge about the network

There are two missing points here:

The gossip information do not contain the current amount of channels, but only the maximum one, so you do not know if the pay is good for the payment (e.g: There is no enough liquidity in one direction)
You can work offline and but the previous point persist

Is there any public discussion somewhere about whats wrong with pathfinding?

I guess the mailing list post and some research paper

ElementsProject / lightning