Reconnection to disconnected peer should not use persistent connection.

[hazrulnizam]

Describe the problem/bug When there is a channel peer that goes offline, ThunderHub will attempt reconnection every start of the hour. Every cycle, ThunderHub will call ten (10) instances of connection attempts, each with persistence set to true. If the peer is offline for an extended amount of time, these persistent connections will accumulate, and eventually crash the lnd backend like in https://github.com/lightningnetwork/lnd/issues/6866.

Since persistent connection will keep retrying every 5 minutes anyway, ThunderHub should set persistence to false instead of true when calling the connect RPC call.

Your environment

• ThunderHub 0.13.15
• Deployment method: Git clone.
• Other relevant environment details: lnd v0.15.0-beta

To Reproduce Steps to reproduce the behavior:

1. Start ThunderHub with loglevel debug
2. Wait until the start of the hour (minute 0 second 0)
3. See log output of both ThunderHub and lnd

Expected behavior Either leave reconnection attempts to lnd to resolve by itself, or, if ThunderHub wants to interfere, use non-persistent connection requests instead of persistent ones.

Actual behavior Every hour, ThunderHub makes ten (10) persistent connect RPC calls to lnd, polluting lnd's persistent connection pool with redundant connections.

Screenshots/Links Example output on the start of the hour:

ThunderHub output:

Aug 30 20:00:00 localhost npm[100]: {
Aug 30 20:00:00 localhost npm[100]:   message: 'Reconnecting to disconnected peers for all nodes.',
Aug 30 20:00:00 localhost npm[100]:   level: 'debug',
Aug 30 20:00:00 localhost npm[100]:   timestamp: '2022-08-30T12:00:00.008Z'
Aug 30 20:00:00 localhost npm[100]: }

lnd output:

2022-08-30 20:00:00.123 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:00.123 [WRN] SRVR: Already have 111 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:00.238 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:00.238 [WRN] SRVR: Already have 112 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:00.454 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:00.454 [WRN] SRVR: Already have 113 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:00.869 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:00.869 [WRN] SRVR: Already have 114 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:01.685 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:01.685 [WRN] SRVR: Already have 115 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:03.300 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:03.300 [WRN] SRVR: Already have 116 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:06.516 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:06.516 [WRN] SRVR: Already have 117 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:12.932 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:12.932 [WRN] SRVR: Already have 118 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:25.746 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:25.746 [WRN] SRVR: Already have 119 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.
2022-08-30 20:00:51.359 [DBG] RPCS: [connectpeer] requested connection to <offline_node_pubkey>@<offline_node_tor_address>.onion:9735
2022-08-30 20:00:51.360 [WRN] SRVR: Already have 120 persistent connection requests for <offline_node_pubkey>@<offline_node_tor_address>.onion:9735, connecting anyway.

[hazrulnizam]

As a workaround, I just removed these lines from the code and rebuild using npm run build.

https://github.com/apotdevin/thunderhub/blob/e2065e1c888508ea780fdb7a6abcbf36532ee11c/src/server/modules/api/bos/bos.resolver.ts#L150-L250

Now, it is no longer spawning new persistent connection requests every hour.

[apotdevin]

Nice catch! Thanks for reporting this, I'll get a fix in

[atomantic]

I was just going to report this. Given that it's been over a year, is this still in progress?

[atomantic]

Ah, I see that you've disabled the cron and referenced this issue. I noticed when I tried to add a peer, it spins up several connection attempts at once, and then keeps stacking them. This may be because my peer connection to this node is failing (tor issue potentially)

This only happens when using thunderhub peer add (not lncli connect).

Entering a peer into the UI and clicking connect once triggers all of these events:

2024-01-03 22:48:18.512 [WRN] SRVR: Already have 35 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:18.620 [WRN] SRVR: Already have 36 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:18.826 [WRN] SRVR: Already have 37 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:19.237 [WRN] SRVR: Already have 38 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:20.043 [WRN] SRVR: Already have 39 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:21.650 [WRN] SRVR: Already have 40 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:24.863 [WRN] SRVR: Already have 41 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.
2024-01-03 22:48:31.277 [WRN] SRVR: Already have 42 persistent connection requests for 02d58859b8fc0e389b7d55a023520bec81aa95c3c91f90a79dcaf7a85a766f27ad@5tygojerixdblky25w3gjs5xbfgisajwwqms7kn3eovpfgn7pfdhzoyd.onion:9735, connecting anyway.

I'll dig in a little deeper later.