CLN master unable to connect to LND node

[grubles]

A change in the past week or two has caused my signet test nodes to be unable to connect to each other.

One is an LND 0.17.0-beta.rc2 and the other is latest CLN master. The only thing that I've changed recently is the CLN node, which I try to track master with. Currently it's running at commit 3f4306eea99c7a61af2fa0132de2d191cf609cec.

LND logs when the CLN node is disconnected:

2024-02-08 13:46:17.734 [INF] PEER: Peer(036e680823a129ac9a2eeb8c43514a5b6cb09ae28e3e698a0e34aaa8865d6cc344): unable to read message from peer: current sid: 124989:1:0 isn't greater than last sid: 162922:1:0
2024-02-08 13:46:17.734 [INF] PEER: Peer(036e680823a129ac9a2eeb8c43514a5b6cb09ae28e3e698a0e34aaa8865d6cc344): disconnecting 036e680823a129ac9a2eeb8c43514a5b6cb09ae28e3e698a0e34aaa8865d6cc344@127.0.0.1:35272, reason: read handler closed

CLN logs while this happens:

2024-02-09T03:09:34.273Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: Splice resume check with local_next_funding: omitted, remote_next_funding: empty, inflights: 0                                                                                                                                                           
2024-02-09T03:09:34.273Z DEBUG   lightningd: Plugin commando returned from custommsg hook call                                                                                         
2024-02-09T03:09:34.273Z DEBUG   lightningd: Calling custommsg hook of plugin peerswap                                                                                                 
2024-02-09T03:09:34.273Z DEBUG   connectd: drain_peer draining subd!                                                                                                                   
2024-02-09T03:09:34.273Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: option_static_remotekey: fields are correct                       
2024-02-09T03:09:34.273Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: next_revocation_number = 654                                      
2024-02-09T03:09:34.273Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-lightningd: peer_disconnect_done  
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-lightningd: Will try reconnect in 128 seconds                                      
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: No upgrade: no next_to_send received
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: option_static_remotekey: fields are correct                       
2024-02-09T03:09:34.274Z DEBUG   connectd: maybe_free_peer freeing peer!                                                                                                               
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#4: channel_gossip: reestablished             
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: No upgrade: no next_to_send received                              
2024-02-09T03:09:34.274Z DEBUG   plugin-funder: Cleaning up inflights for peer id 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923                
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: billboard perm: Reconnected, and reestablished.
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#6: channel_gossip: reestablished             
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: billboard perm: Reconnected, and reestablished.                   
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: billboard: Channel ready for use.              
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: billboard: Channel ready for use.
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: billboard: Channel ready for use.
2024-02-09T03:09:34.274Z DEBUG   035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: billboard: Channel ready for use. 
2024-02-09T03:09:34.274Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#4: Peer connection lost            
2024-02-09T03:09:34.275Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#4: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)                                                                           
2024-02-09T03:09:34.275Z DEBUG   lightningd: Plugin peerswap returned from custommsg hook call                                                                                         
2024-02-09T03:09:34.275Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-channeld-chan#6: Peer connection lost                                              
2024-02-09T03:09:34.275Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#6: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)

[rustyrussell]

Weird! IO level logging might help here, but we would still be guessing what LND gets upset by...

[grubles]

Here is an IO level log for the first 1 minute or so of starting CLN:

logs.tar.gz

[grubles]

I noticed this happening again only after I had restarted the CLN node. This time with LND 0.17.4-beta and CLN v24.02.2.

[cdecker]

Turns out it is a gossip_query_reply in which the sorting order was expected by lnd but potentially CLN did not sort them?

https://github.com/lightningnetwork/lnd/blob/2ccf58e2afa1fc3718f69d20af5aa3a79723a206/lnwire/query_short_chan_ids.go#L21-L32