track: Receiving payments is slow when we have blockchain lag

[danielgranhao]

We've noticed one of our nodes consistently taking longer to receive payments when compared to the rest. This one takes around 1 minute while others take under 10 seconds.

The node ID is 035458b2dd3e2a89213f6ea4cb205463c6d1c3bbbafd30d77ffa1875e957ae1382

[cdecker]

Ok, looking into session_id=7197154242203648 from 2024-05-17T10:41:29+02:00

Timeline

• 2024-05-17T10:41:29 Node scheduled
  • 2 channels, no HTLCs pending
• 2024-05-17T10:41:32 Node startup complete
• 2024-05-17T10:41:35 Signer 0 connected
• 2024-05-17T10:41:36 Gossip sync complete
• 2024-05-17T10:41:40 Both channels are correctly re-established
• 2024-05-17T10:42:46 Signer 1 connected
• 2024-05-17T10:43:44 Signer 0 dies
• 2024-05-17T10:43:48 Receiving an incoming HTLC
• 2024-05-17T10:43:50 Blockchain sync not completed yet (843793), deferring the commit
• 2024-05-17T10:44:24 Finally synced with the chain, committing changes now that we can be sure the channel is still alive

Blockchain sync in progress from block 843708 to 843837.

Analysis

So the root cause for this 30 second receive is the blockchain lag that the node had at that time (129 blocks). Obviously the payment itself may have taken longer to find a route and actually route the payment to us, but that's the part we have control over.

Resolution

This is the counterpart to https://github.com/ElementsProject/lightning/pull/7190 which we just release a couple of weeks ago. We are planning to tackle this in two ways:

• Ensure the blocklag is reduced, resulting in a reduction of time until we can acknowledge the HTLC
• Free us from having to fully sync altogether by splitting the final CLTV into two components: lag-allowance and security parameter. Lag allowance allows us to accept HTLCs before being synced by extending the security parameter by a buffer. Should we notice the channel closed in the meantime, we can still punish the counterparty by using the security margin.

This may take a bit, as it requires going all the way up the stack to CLN to implement this, but it should be doable.

[cdecker]

Both regions are now back in sync with the blockchain, with individual stragglers due to different block processing speeds, depending on number of addresses we're checking for.