Slow payments

[yaslama]

Paying from/to GL and comparing the time until the payment is settled to a payment with the same path except using a regular CLN node instead of GL node. The payment using GL is more than 2 seconds slower than the payment using CLN.

[cdecker]

We briefly discussed this in chat yesterday, and I think the vast majority is due to network latencies. The nodes in prod are currently located in SF (GCP region us-west2), while the client/signer is in Israel, and the LSP node is in Paris.

This results in average RTTs of:

• node <-> signer: 200ms
• node <-> LSP: 160ms

I'm assuming that you tested with a stock CLN node running in the same rough region as the signer or the LSP, so we can drop those to ~5ms for one direction, and ~50ms for the other latency.

Now, in the best case scenario a payment consists of 3 RTT over the LSP <-> node link

• => update_add_htlc + commitment_signed
• <= revoke_and_ack + commitment_signed
• => revoke_and_ack
• <= update_fulfill_htlc + commitment_signed
• => revoke_and_ack + commmitment_signed
• <= revoke_and_ack

Each of the commitment_signed sent, and revoke_and_ack sent and received then triggers also a request to the signer to sign off on the state changes (hsmd_validate_commitment_tx, hsmd_validate_revocation).

3 * 160ms + 6 * 200ms = 480ms + 1200ms = 1680

There may be a bit more overhead due to the verification, which the stock CLN signer doesn't have to perform, but the network latencies are likely the far bigger contributor.

While we will always be slower than a local CLN signer that doesn't have to adhere to speed of light concerns, we have a number of improvements that we plan to add eventually:

• Geo-colocation: the most obvious fix to address the latency caused by long distances is to co-locate the nodes closer with the signer and/or the LSP. Given the above breakdown it'd make sense to migrate nodes whose clients mostly connect from EU to the EU, US to US, and Asia to Asia. However this migration capability may take some time to be implemented and we need a sufficiently large user-base in each of these geos first to warrant such an optimization.
• Anycast and using the GCP premium network: we can locate the proxies in a number of geos, have them advertise via an anycast IP address, so that clients connect to their closest point-of-presence that runs a proxy, and from there on all communication is handled between services the GL team controls, allowing us to use the direct fiber links from GCP. While I don't expect this to be as big an improvement as the co-location, it could shave off a couple ms and get us closer to the speed of light limit. According to GCP's inter-regional latency tracking that would be 156ms between eu-west6 and us-west2, but we'd still have some overhead of getting from user to the proxy, so that'd be a lower bound anyway.

[kingonly]

Currently, sending payments takes 15-50s even for partners in the EU. A few months ago I was able to pay in 7s, but that's no longer the case. Weirdly, even w/o waiting for sync, initial payments take much longer than subsequent payments.

How do we deal with payment latency?