Closed devastgh closed 1 year ago
I'm his peer who opened a regular channel to him (we just had a stuck htlc) . I have the dual-fund experimental flag on. I'm not sure if I opened with "fundchannel" or "multifundchannel". Now I have 2 records in "channels". One has the status of AWAITING_UNILATERAL, and another had CHANNELD_NORMAL. As you can see this AWAITING_UNILATERAL channel is kind of stuck on the listpeers. The tx:1d423ddaae344e4f24cd536af9cda7328b74e14b157308ad761ff49c4a218c85 , never get broadcasted, I think, not sure how it got in there in the first place.
{
"peers": [
{
"id": "03xx",
"connected": false,
"channels": [
{
"state": "AWAITING_UNILATERAL",
"scratch_txid": "e596b3b3d47a68049490d3e2ac18c90bfcf3da11a85c689711961a0bc007a2c0",
"last_tx_fee": "614000msat",
"last_tx_fee_msat": "614000msat",
"feerate": {
"perkw": 253,
"perkb": 1012
},
"channel_id": "5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc",
"funding_txid": "1d423ddaae344e4f24cd536af9cda7328b74e14b157308ad761ff49c4a218c85",
"funding_outnum": 1,
"initial_feerate": "253perkw",
"last_feerate": "253perkw",
"next_feerate": "256perkw",
"inflight": [
{
"funding_txid": "1d423ddaae344e4f24cd536af9cda7328b74e14b157308ad761ff49c4a218c85",
"funding_outnum": 1,
"feerate": "253perkw",
"total_funding_msat": "10000000000msat",
"our_funding_msat": "10000000000msat",
"scratch_txid": "e596b3b3d47a68049490d3e2ac18c90bfcf3da11a85c689711961a0bc007a2c0"
}
],
"close_to_addr": "xxxx",
"close_to": "xxxx",
"private": false,
"opener": "local",
"closer": null,
"features": [
"option_static_remotekey",
"option_anchor_outputs"
],
"funding_allocation_msat": {
"xxxx": 0,
"xxxx": 10000000000
},
"funding_msat": {
"xxxx": "0msat",
"xxxx": "10000000000msat"
},
"funding": {
"local_msat": "10000000000msat",
"remote_msat": "0msat",
"pushed_msat": "0msat"
},
"msatoshi_to_us": 10000000000,
"to_us_msat": "10000000000msat",
"msatoshi_to_us_min": 10000000000,
"min_to_us_msat": "10000000000msat",
"msatoshi_to_us_max": 10000000000,
"max_to_us_msat": "10000000000msat",
"msatoshi_total": 10000000000,
"total_msat": "10000000000msat",
"fee_base_msat": "0msat",
"fee_proportional_millionths": 3000,
"dust_limit_satoshis": 546,
"dust_limit_msat": "546000msat",
"max_htlc_value_in_flight_msat": 18446744073709551615,
"max_total_htlc_in_msat": "18446744073709551615msat",
"their_channel_reserve_satoshis": 100000,
"their_reserve_msat": "100000000msat",
"our_channel_reserve_satoshis": 100000,
"our_reserve_msat": "100000000msat",
"spendable_msatoshi": 4294967295,
"spendable_msat": "4294967295msat",
"receivable_msatoshi": 0,
"receivable_msat": "0msat",
"htlc_minimum_msat": 0,
"minimum_htlc_in_msat": "0msat",
"minimum_htlc_out_msat": "0msat",
"maximum_htlc_out_msat": "9900000000msat",
"their_to_self_delay": 144,
"our_to_self_delay": 2016,
"max_accepted_htlcs": 30,
"state_changes": [
{
"timestamp": "2022-05-14T18:49:26.066Z",
"old_state": "DUALOPEND_OPEN_INIT",
"new_state": "AWAITING_UNILATERAL",
"cause": "protocol",
"message": "dualopend: received ERROR error channel 5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc: Unknown channel for
WIRE_CHANNEL_REESTABLISH"
}
],
"status": [
"AWAITING_UNILATERAL:Will attempt reconnect in 240 seconds"
],
"in_payments_offered": 0,
"in_msatoshi_offered": 0,
"in_offered_msat": "0msat",
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"in_fulfilled_msat": "0msat",
"out_payments_offered": 0,
"out_msatoshi_offered": 0,
"out_offered_msat": "0msat",
"out_payments_fulfilled": 0,
"out_msatoshi_fulfilled": 0,
"out_fulfilled_msat": "0msat",
"htlcs": []
},
{
"state": "ONCHAIN",
"scratch_txid": "d1bbf79693b2ae2606ef900ebc3a15f50a7564a534ca6744fe7c682c5c840209",
"last_tx_fee": "4433000msat",
"last_tx_fee_msat": "4433000msat",
"feerate": {
"perkw": 3419,
"perkb": 13676
},
"owner": "onchaind",
"short_channel_id": "733764x416x0",
"direction": 0,
"channel_id": "xxxxx",
"funding_txid": "xxxx",
"funding_outnum": 0,
"close_to_addr": "xxx",
"close_to": "xxxx",
"private": false,
"opener": "local",
"closer": "remote",
"features": [
"option_static_remotekey",
"option_anchor_outputs"
],
"funding_allocation_msat": {
"xxxx": 0,
"xxxx": 10000000000
},
"funding_msat": {
"xxx": "0msat",
"xxxx": "10000000000msat"
},
"funding": {
"local_msat": "10000000000msat",
"remote_msat": "0msat",
"pushed_msat": "0msat"
},
"msatoshi_to_us": 7840338812,
"to_us_msat": "7840338812msat",
"msatoshi_to_us_min": 7840338812,
"min_to_us_msat": "7840338812msat",
"msatoshi_to_us_max": 10000000000,
"max_to_us_msat": "10000000000msat",
"msatoshi_total": 10000000000,
"total_msat": "10000000000msat",
"fee_base_msat": "0msat",
"fee_proportional_millionths": 40,
"dust_limit_satoshis": 546,
"dust_limit_msat": "546000msat",
"max_htlc_value_in_flight_msat": 18446744073709551615,
"max_total_htlc_in_msat": "18446744073709551615msat",
"their_channel_reserve_satoshis": 100000,
"their_reserve_msat": "100000000msat",
"our_channel_reserve_satoshis": 100000,
"our_reserve_msat": "100000000msat",
"spendable_msatoshi": 4294967295,
"spendable_msat": "4294967295msat",
"receivable_msatoshi": 1968657733,
"receivable_msat": "1968657733msat",
"htlc_minimum_msat": 0,
"minimum_htlc_in_msat": "0msat",
"minimum_htlc_out_msat": "0msat",
"maximum_htlc_out_msat": "9900000000msat",
"their_to_self_delay": 144,
"our_to_self_delay": 2016,
"max_accepted_htlcs": 30,
"state_changes": [
{
"timestamp": "2022-04-27T08:10:14.444Z",
"old_state": "DUALOPEND_OPEN_INIT",
"new_state": "DUALOPEND_AWAITING_LOCKIN",
"cause": "user",
"message": "Sigs exchanged, waiting for lock-in"
},
{
"timestamp": "2022-04-27T08:43:06.910Z",
"old_state": "DUALOPEND_AWAITING_LOCKIN",
"new_state": "CHANNELD_NORMAL",
"cause": "user",
"message": "Lockin complete"
},
{
"timestamp": "2022-05-15T18:31:29.579Z",
"old_state": "CHANNELD_NORMAL",
"new_state": "AWAITING_UNILATERAL",
"cause": "onchain",
"message": "Funding transaction spent"
},
{
"timestamp": "2022-05-15T18:31:29.579Z",
"old_state": "AWAITING_UNILATERAL",
"new_state": "FUNDING_SPEND_SEEN",
"cause": "onchain",
"message": "Onchain funding spend"
},
{
"timestamp": "2022-05-15T18:31:30.043Z",
"old_state": "FUNDING_SPEND_SEEN",
"new_state": "ONCHAIN",
"cause": "onchain",
"message": "Onchain init reply"
}
],
"status": [
"ONCHAIN:Tracking their unilateral close",
"ONCHAIN:All outputs resolved: waiting 95 more blocks before forgetting channel"
],
"in_payments_offered": 7,
"in_msatoshi_offered": 2871834014,
"in_offered_msat": "2871834014msat",
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"in_fulfilled_msat": "0msat",
"out_payments_offered": 40,
"out_msatoshi_offered": 10497219824,
"out_offered_msat": "10497219824msat",
"out_payments_fulfilled": 9,
"out_msatoshi_fulfilled": 2159661188,
"out_fulfilled_msat": "2159661188msat",
"htlcs": [
{
"direction": "in",
"id": 6,
"msatoshi": 91003455,
"amount_msat": "91003455msat",
"expiry": 736509,
"payment_hash": "5ca9c3c94c0a3dffdf7dcca7aae5c2ac191eb82d7cd7ef578327d7e73702b791",
"state": "SENT_REMOVE_COMMIT"
}
]
}
]
}
]
From lnlog
2022-04-27T08:09:26.303Z INFO 03xx-chan#22181: Unsaved peer failed. Disconnecting and deleting channel.
2022-04-27T08:10:14.443Z INFO 03xx-chan#22182: State changed from DUALOPEND_OPEN_INIT to DUALOPEND_AWAITING_LOCKIN
2022-04-27T08:43:06.908Z INFO 03xx-chan#22182: State changed from DUALOPEND_AWAITING_LOCKIN to CHANNELD_NORMAL
2022-04-27T08:54:55.042Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T08:54:55.042Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T08:54:55.043Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T08:54:55.043Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-27T08:55:23.595Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T08:55:23.595Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-27T08:55:23.596Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T08:55:23.596Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T11:55:12.771Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T11:55:12.771Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T11:55:12.772Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T11:55:12.772Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-27T12:05:38.152Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T12:05:38.152Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T12:05:38.153Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T12:05:38.153Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-27T14:13:19.255Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T14:13:19.256Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-27T14:13:19.256Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T14:13:19.256Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T14:17:00.401Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-04-27T14:17:00.402Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-27T14:17:00.402Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-04-27T14:17:00.402Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-28T04:44:14.352Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-28T04:44:14.352Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-29T20:07:25.475Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-04-29T20:07:25.475Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-04-30T04:05:35.932Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: Reconnected
2022-04-30T04:05:35.932Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: Reconnected
2022-04-30T04:05:40.946Z **BROKEN** 03xx-connectd: Peer did not close, forcing close
2022-05-01T21:59:21.258Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-05-01T21:59:21.258Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-05-02T16:22:23.221Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-05-02T16:22:23.221Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-05-02T16:37:47.032Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-05-02T16:37:47.032Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-05-02T16:41:12.783Z INFO 03xx-dualopend-chan#22180: Peer connection lost
2022-05-02T16:41:12.783Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: dualopend: Owning subdaemon dualopend died (62208)
2022-05-02T16:41:12.891Z INFO 03xx-channeld-chan#22182: Peer connection lost
2022-05-02T16:41:12.891Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: channeld: Owning subdaemon channeld died (62208)
2022-05-02T16:41:25.614Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: Reconnected
2022-05-02T16:41:25.614Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: Reconnected
2022-05-02T16:41:31.696Z INFO 03xx-chan#22180: Peer transient failure in DUALOPEND_OPEN_INIT: Reconnected
2022-05-02T16:41:31.696Z INFO 03xx-chan#22182: Peer transient failure in CHANNELD_NORMAL: Reconnected
2022-05-02T16:41:33.017Z **BROKEN** 03xx-connectd: Peer did not close, forcing close
2022-05-02T21:01:48.513Z INFO 03xx-dualopend-chan#22180: Peer connection lost
And keeps on showing
**BROKEN** 03xx-connectd: Peer did not close, forcing close
Sorry, it was a few weeks ago I'm not sure if I was on rc5 or 0.11.0.
Turns out the limbo channel prevents my node to connect to his. When I connect the connect command never returns and I get this in my log,
2022-05-25T09:20:32.298Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: Connected out, starting crypto
2022-05-25T09:20:32.907Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: Connect OUT
2022-05-25T09:20:32.907Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_out WIRE_INIT
2022-05-25T09:20:33.529Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_in WIRE_INIT
2022-05-25T09:20:33.529Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-gossipd: seeker: disabling gossip
2022-05-25T09:20:33.530Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-chan#22180: Peer has reconnected, state AWAITING_UNILATERAL: telling connectd to make active
2022-05-25T09:20:33.530Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: Handed peer, entering loop
2022-05-25T09:20:33.531Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_out WIRE_GOSSIP_TIMESTAMP_FILTER
2022-05-25T09:20:33.531Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-lightningd: Telling connectd to send error 00115fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc00194177616974696e6720756e696c61746572616c20636c6f7365
2022-05-25T09:20:33.532Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: destroy_subd: 1 subds, to_peer conn 0x56520b87f008, read_to_die = 0
2022-05-25T09:20:33.532Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_out WIRE_ERROR
2022-05-25T09:20:33.532Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_out WIRE_ERROR
2022-05-25T09:20:33.864Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-connectd: peer_conn_closed
2022-05-25T09:20:33.864Z DEBUG 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-lightningd: peer_disconnect_done
2022-05-25T09:20:33.865Z DEBUG plugin-funder: Cleaning up inflights for peer id 03xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
How can I clear this limbo channel out of my db ? I wonder, since I'm unable to connect to him, maybe it's a reason that makes the limbo channel stay on forever.
Looks like that this problem is no longer reproducible https://github.com/ElementsProject/lightning/issues/6255
So I am closing this and feel free to reopen it if this happens on new releases
Issue and Steps to Reproduce
Situation: My cln version: 0.11 . Both me, and my peer had experimental dual fund enabled. My peer opened a channel to me, that was not a dual-funded channel, only he put funding into it. I had no liquidity ads present. listpeers for that peer had 2 parts in it for me: one owner:dualopend and one owner:channeld. Once the channel got established onchain, it started working properly, owner:channeld status was established, announced, the channel is working fine. However, owner:dualopend did not disappear and was stuck in init, for 1-2 weeks. After a node restart, owner:dualopend part disappeared, however my log is now full with
Unknown channel 5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc for WIRE_CHANNEL_REESTABLISH
.Relevant part of my logs: when the channel was established, everything looking fine:
After that i started getting this in the log:
In cln getinfo my num_active channels was the correct number (the channel for this peer was counted here), however i had one num_pending_channel, i guess this was the owner:dualopend part that was stuck in init. Ride the lightning showed that the channel was stuck in dual open initialized. The owner:dualopend that was stuck had an unique channel id, different than the owner:channeld. After a restart it's trying to reestablish the channel id, that was in the owner:channeld part, that was never online, every minute or so:
2022-05-13T07:06:30.585Z UNUSUAL xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-lightningd: Unknown channel 5fa36123c25df987af2282ee39384d8ef0f468f270c569fa596e55b1c6d37ddc for WIRE_CHANNEL_REESTABLISH
After consulting with niftynei i'm opening this issue. I can not provide the old listpeers output that had owner:dualopend, since it's not there anymore. Just my guess is: cln expected me to put funding into the channel, and it was waiting for an onchain transaction. If i remember correctly, i had a funding txid in owner:dualopend, that was actually never done, since i did not put any funding into the channel.