Bug: core lightning nodes fail to connect as source

[AlessandroZanatta]

Describe the bug Trying to connect stock Core Lightning nodes fails with the following: Unable to open the channel lightningd 400: Unable to connect, no address known for peer

Partial log from docker logs:

2023-08-12T09:45:26.451Z DEBUG   hsmd: Client: Received message 1 from client
2023-08-12T09:45:26.451Z DEBUG   connectd: Responder: handshake failed act_one_responder2:931
2023-08-12T09:45:26.451Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Failed connected out: All addresses failed: 0.0.0.0:9735: Cryptographic handshake: peer closed connection (wrong key?). 
2023-08-12T09:45:26.451Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-lightningd: Not reconnecting: no channels
2023-08-12T09:45:26.451Z UNUSUAL plugin-plugin.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"All addresses failed: 0.0.0.0:9735: Cryptographic handshake: peer closed connection (wrong key?). \",\"code\":401,\"fullType\":\"lightning\"}
2023-08-12T09:45:26.461Z DEBUG   plugin-spenderp: mfc 51: multiconnect.
2023-08-12T09:45:26.461Z DEBUG   plugin-spenderp: mfc 51, dest 0: connect 03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead.
2023-08-12T09:45:26.461Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for opening hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for mutual_close hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for unilateral_close hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for delayed_to_us hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for htlc_resolution hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for penalty hit floor 253
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... polled feerate estimate for min_acceptable (125) smoothed to 248 (alpha=0.04)
2023-08-12T09:45:28.279Z DEBUG   lightningd: ... feerate estimate for min_acceptable hit floor 253
2023-08-12T09:45:28.792Z DEBUG   gossipd: seeker: no peers, waiting
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Resolving ln1q0x8d3v32yg05hp90m3nyg4dzj949udy4vxm6lrv8rpkxxslta026yp2tpu.lseed.bitcoinstats.com
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Could not resolve ln1q0x8d3v32yg05hp90m3nyg4dzj949udy4vxm6lrv8rpkxxslta026yp2tpu.lseed.bitcoinstats.com
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Resolving ln1q0x8d3v32yg05hp90m3nyg4dzj949udy4vxm6lrv8rpkxxslta026yp2tpu.lseed.darosior.ninja
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Could not resolve ln1q0x8d3v32yg05hp90m3nyg4dzj949udy4vxm6lrv8rpkxxslta026yp2tpu.lseed.darosior.ninja
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-connectd: Failed connected out: Unable to connect, no address known for peer
2023-08-12T09:45:28.822Z DEBUG   03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead-lightningd: Not reconnecting: no channels
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51, dest 0: failed! connect 03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead: {\"code\":400,\"message\":\"Unable to connect, no address known for peer\"}.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: multiconnect done.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: trying redo despite 'connect' failure (Unable to connect, no address known for peer); will cleanup for now.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: cleanup!
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: Filtering destinations.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51, dest 0: failed.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: 1 destinations failed, failing.
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: cleanup!
2023-08-12T09:45:28.822Z DEBUG   plugin-spenderp: mfc 51: cleanup done, finishing command.
2023-08-12T09:45:28.823Z UNUSUAL plugin-plugin.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Unable to connect, no address known for peer\",\"code\":400,\"data\":{\"id\":\"03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead\",\"method\":\"connect\"},\"fullType\":\"lightning\"}

Trying to connect (as a source) another type of node (e.g. LND) works. However, using lightning core as a source always fails. I've tested multiple version of lightning core, and they all seem to be affected.

To Reproduce Steps to reproduce the behavior:

1. Create a network with a single bitcoind node and two (or more) core lightning nodes
2. Start the network
3. Try to connect the two core lightning nodes
4. Observe failure of channel creation

Expected behavior A channel is created between the two nodes.

Screenshots

Desktop:

• OS: Arch Linux (rolling)
• Polar Version: v2.0.0
• Docker Version: v24.0.5
• Docker Compose Version: v2.20.3

[AlessandroZanatta]

Update:

I've tested Polar on a freshly installed instance of Ubuntu 23.04 (installed from the provided .deb). The problem persists. The versions reported above are all the same, with the exception of docker compose (v2.20.2 instead of v2.20.3).

I've read in the patch notes that support for docker compose (plugin or the deprecated docker-compose?) v2 has been added in the latest release. Could that be the issue? This line seems to hint to some issue regarding networking. 2023-08-12T09:45:28.823Z UNUSUAL plugin-plugin.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"Unable to connect, no address known for peer\",\"code\":400,\"data\":{\"id\":\"03cc76c5915110fa5c257ee33222ad148b52f1a4ab0dbd7c6c38c3631a1f5f5ead\",\"method\":\"connect\"},\"fullType\":\"lightning\"}

A previous one, however, seem to hint to something else entirely: 2023-08-12T09:45:26.451Z UNUSUAL plugin-plugin.js: {\"type\":\"lightning\",\"name\":\"LightningError\",\"message\":\"All addresses failed: 0.0.0.0:9735: Cryptographic handshake: peer closed connection (wrong key?). \",\"code\":401,\"fullType\":\"lightning\"}

[bshramin]

I have the same issue. Maybe we can make the title of this issue more descriptive.

[jamaljsr]

Hey yall, thanks for the bug report. I believe the root cause could be the same as described in #747. I'll confirm this week and get a patch release out if that is indeed the issue.

[AlessandroZanatta]

I have the same issue. Maybe we can make the title of this issue more descriptive.

You're right. I've totally forgotten to add a title to the bug, sorry about that.

[jamaljsr]

This is fixed by #762