Outgoing payments failing with "Ran out of routes after..."

[danielgranhao]

Receiving works, but sending payments always fails with a message such as "Ran out of routes to try after 61 attempts: see paystatus".

Example of invoice I tried to pay: lnbc200u1pnyx2fvpp5633ln90ffuyul5gxtfklch8qx8n6pj3zs6c7yjfgxq93h38cn68qdqqcqzzsxqyz5vqsp5wxhcsg8ug0f4zqnvw9z42dr7ud4ezxwu849f8rnwf0ndcjwlr0es9qyyssqapnksmzu2dfh0uel7nun9a4qdrhk9echp26ge5erlge92hhj5vczerwmh0hpnkafj488yh4j5599drcw7h8m9enk8p6j03s43thay6gp3d575q

Node ID: 02605ee12e57d9366391f963550e3895ad8f2c9ab492de67d5eb2fe4c05432d8b7

[cdecker]

From the logs:

• pay call at 20240503T090357Z: Waits for channels to be re-established, fails, and then complains about We don't have any channels (usually an indicator that the private gossip is missing)
• 20240503T090546: pay call in the same session as above, also fails to wait for channel re-establish, also runs into the We don't have any channels issue
• 20240507T081417: pay call, new session, fails to wait, thus runs into the We don't have any channel issue
• 20240507T111930: pay call, new session, fails to await channel readiness, but then uses the routehint to try anyway because we can't find ourselves in the graph (definitely missing gossip) and then spirals trying to split and find a path for the smaller splits over and over again
• 20240507T112018: pay call, same session, fails again, then grinds the amounts to the split limit, never finding a path
• 20240507T162419: waits, fails and ends with We don't have any channels
• 20240508T151993: Does not have to wait because node is already reconnected before pay call. Correctly computes the route 824204x2799x0 -> 818874x1478x0 -> 820200x636x1. HTLC is fulfilled after 20s, pay call succeeds
• 20240508T152006: same session, path computed 824204x2799x0 -> 818874x1478x0 -> 837626x2102x0, HTLC fulfilled after 12s
• 20240508T152327: same session, path 824204x2799x0 -> 818874x1478x0 -> 837626x2102x0, fulfilled after 32s
• 20240514T082814Z: waits, fails to find its own channels, then claims we don't have any channels
• 20240514T083014Z: same as above since it is the same session
• 20240514T084636: new session, this time we try, but `Could not locate ourselves in the network, so we rely on direct attempts working, which fails and we end up in the grinding situation again
• 20240514T084650: same session, same outcome

So as far as we can tell all of these issues are related to the private gossip not working as expected. The good news is that it'll be automatically fixed with the upcoming v24.02gl1 version, which internalizes the handling of private gossip, instead of managing it externally.

[danielgranhao]

Thanks @cdecker. Any timeline already defined for the release of that version?

[cdecker]

It is going out as soon as possible, but first we want to gain an overview over the issues, what issues will be addressed by that upgrade, so we don't end in a situation where we can't debug issues any more (causing them to remain open indefinitely or being closed as non-reproducible, neither of which is a good outcome).

[cdecker]

With v24.02gl1 rolling out I think we can close this.