search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.63k stars 2.07k forks source link

[bug]: Opening Channels not broadcast #8987

Open ostermayer opened 1 month ago

ostermayer commented 1 month ago

Background

After closing approximately 10 channels and then attempting to open 4 channels using the opening channels were not broadcast Describe your issue here. Full node backed hosted instance via voltage (professional node)

Your environment

voltage

Steps to reproduce

total_limbo_balance": "11500731",
    "pending_open_channels": [
        {
            "channel": {
                "remote_node_pub": "0288be11d147e1525f7f234f304b094d6627d2c70f3313d7ba3696887b261c4447",
                "channel_point": "a4d8b54947f935927688be00f5e012d4d3d9e78f715879f31d922c4092b40b78:1",
                "capacity": "32788259",
                "local_balance": "32786472",
                "remote_balance": "0",
                "local_chan_reserve_sat": "327882",
                "remote_chan_reserve_sat": "327882",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false,
                "memo": ""
            },
            "commit_fee": "1127",
            "commit_weight": "772",
            "fee_per_kw": "1003",
            "funding_expiry_blocks": 940
        },
        {
            "channel": {
                "remote_node_pub": "02e9046555a9665145b0dbd7f135744598418df7d61d3660659641886ef1274844",
                "channel_point": "aa2dba58276e2454704b8981cd5e57040f4eb3722fd524756a819cfaec1b41b1:1",
                "capacity": "34804050",
                "local_balance": "34802263",
                "remote_balance": "0",
                "local_chan_reserve_sat": "34804",
                "remote_chan_reserve_sat": "348040",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false,
                "memo": ""
            },
            "commit_fee": "1127",
            "commit_weight": "772",
            "fee_per_kw": "1003",
            "funding_expiry_blocks": 944
        },
        {
            "channel": {
                "remote_node_pub": "030b5d1a2f2502eb14f7dfccd6de8d227cc533fe14214c80ecb5dd9bd6ef9dc8e9",
                "channel_point": "de174045436998c71283b3cc3153731e11cecb1d51f3f7994bd84314f0cbde05:1",
                "capacity": "34825334",
                "local_balance": "34823547",
                "remote_balance": "0",
                "local_chan_reserve_sat": "348253",
                "remote_chan_reserve_sat": "348253",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false,
                "memo": ""
            },
            "commit_fee": "1127",
            "commit_weight": "772",
            "fee_per_kw": "1003",
            "funding_expiry_blocks": 953
        },
        {
            "channel": {
                "remote_node_pub": "0324ba2392e25bff76abd0b1f7e4b53b5f82aa53fddc3419b051b6c801db9e2247",
                "channel_point": "56b713b91700c581253ea638e788378414229818d3ab24dd369258f6d3d7b563:0",
                "capacity": "25298299",
                "local_balance": "25296512",
                "remote_balance": "0",
                "local_chan_reserve_sat": "252982",
                "remote_chan_reserve_sat": "252982",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false,
                "memo": ""
            },
            "commit_fee": "1127",
            "commit_weight": "772",
            "fee_per_kw": "1003",
            "funding_expiry_blocks": 934
        }

Expected behaviour

Opened channels would broadcast

Actual behaviour

newly opened channels did not broadcast and the utxo set used for the channel opening seemed to be used more than once for the channel opens as the balance on the node when accounting for the "pending channels" exceeded the original balance on the node. this leads me to think the utxo were attempted to be "double spent" by LND. Then after performing a rescan I was able to move the comitted utxos off the node onchain. The channels that are pending which do not have the utxos anymore are still in pending status.

Then via LND terminal web i chose to force close the channels that never opened and they are now stuck in "force close status pending"

EDIT: formatting for readability

ostermayer commented 1 month ago

seems to be similar to this bug https://github.com/lightningnetwork/lnd/issues/6048

hieblmi commented 1 month ago

Thanks for reporting this issue. If you are absolutely sure that all your on-chain funds are recovered from these opening attempts you can abandon these channels via lncli to clean up the pending channel states on your node.

ProofOfKeags commented 1 month ago

This suggests to me that we don't properly have resource/utxo locking during the channel funding process.

ostermayer commented 1 month ago

yes my peer who has the other side of this non broadcast channel has this message from their node

2024-08-07 18:20:02,287 WARN f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - force-closing channel at user request 2024-08-07 18:20:02,289 INFO f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - using finalScriptPubkey=ByteVector(22 bytes, 0x00142af5dff4afd2f4bf7624a73ff15fd050db65dd2a) 2024-08-07 18:20:02,295 ERROR f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - force-closing with fundingIndex=0 2024-08-07 18:20:02,313 INFO f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - tx generation skipped: desc=local-main-delayed reason: output not found (probably trimmed) 2024-08-07 18:20:02,327 INFO f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - tx generation skipped: desc=local-anchor reason: output not found (probably trimmed) 2024-08-07 18:20:02,328 INFO f.a.e.c.fsm.Channel n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - tx generation success: desc=remote-anchor txid=db9925c5686c1e067b5f133d33c20d62269cfff6a13b78c50c0cc4e6eb9a375c amount=0 sat tx=02000000011abfb5402be8b542baae6fcedf2e9946a088dbc368dc6afaad28171d48b1f7c90000000000000000000000000000 2024-08-07 18:20:02,332 INFO f.a.e.c.p.TxPublisher n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - publishing commit-tx txid=c9f7b1481d1728adfa6adc68c3db88a046992edfce6faeba42b5e82b40b5bf1a spending aa2dba58276e2454704b8981cd5e57040f4eb3722fd524756a819cfaec1b41b1:1 with id=58a9953f-ed63-4dda-8735-78b25d1312ef (0 other attempts) 2024-08-07 18:20:02,333 INFO f.a.e.i.Peer n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - OUT msg=Error(b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab,ByteVector(19 bytes, 0x666f72636564206c6f63616c20636f6d6d6974),TlvStream(Set(),Set())) 2024-08-07 18:20:02,333 INFO f.a.eclair.Diagnostics n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab - OUT msg=Error(b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab,ByteVector(19 bytes, 0x666f72636564206c6f63616c20636f6d6d6974),TlvStream(Set(),Set())) 2024-08-07 18:20:02,354 INFO f.a.e.c.p.MempoolTxMonitor n:03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9 c:b1411becfa9c816a7524d52f72b34e0f04575ecd81894b7054246e2758ba2dab t:58a9953f - could not publish tx: one of our inputs cannot be found

ostermayer commented 1 month ago

in my limited understanding, it seems that if there was a transaction not broadcast that corresponds to a pending channel and that utxo does not exist anymore then the pending channel should be purged from LND database.

ostermayer commented 1 month ago

Thanks for reporting this issue. If you are absolutely sure that all your on-chain funds are recovered from these opening attempts you can abandon these channels via lncli to clean up the pending channel states on your node.

yes, still somewhat nervous on doing "abandon" and hopeful that a future LND version will cleanup the database as I also have a channel that was closed and has many many confirmations but is still listed as "closing" and has been for some time