search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.66k stars 2.07k forks source link

[bug]: Unknown Unconfirmed Balance #7104

Open WalterSmuts opened 1 year ago

WalterSmuts commented 1 year ago

Background

I folks, am running LND as a router. I have unconfirmed balance and not really sure what's happening. Haven't made any channel opens or blockchain payments since way before this issue came up. So I think it must be related to lnd closing a channel.

Here's the relevant data:

$ lncli walletbalance | grep uncon
    "unconfirmed_balance": "1051070",
            "unconfirmed_balance": "1051070"
$ lncli listunspent --unconfirmed_only
{
    "utxos": [
        {
            "address_type": 4,
            "address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
            "amount_sat": 1051070,
            "pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
            "outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
            "confirmations": 0
        }
    ]
}
$ lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

The relevant transaction doesn't appear on public trackers. Any idea what happened and how to fix?

Your environment

ellemouton commented 1 year ago

Hi @WalterSmuts! Thanks for opening the issue :)

On first glance, this looks very similar to #6367, #6231, #6498, #6434

Could you perhaps grep your logs for the transaction ID? (3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97)

WalterSmuts commented 1 year ago 
$ journalctl -u lnd | grep 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97
Oct 24 10:36:20 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:20.526 [INF] SWPR: Creating sweep transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97 for 2 inputs (1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0 (CommitmentAnchor), 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 (TaprootPubKeySpend)) using 1803 sat/kw, tx_weight=725, tx_fee=0.00002501 BTC, parents_count=1, parents_fee=0.00001128 BTC, parents_weight=1288
Oct 24 10:36:26 [REDACTED_IP] lnd[265812]: 2022-10-24 10:36:26.733 [INF] LNWL: Inserting unconfirmed transaction 3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97
ellemouton commented 1 year ago

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Do you recall trying to close this channel? Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID)

WalterSmuts commented 1 year ago

interesting. Is your channel with outpoint 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0 still listed as open in your listchannels output?

Nope, don't think so.

lncli listchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      0       0       0
$ lncli closedchannels  | grep 67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d | wc
      1       2     101

Do you recall trying to close this channel?

Definitely not personally. Could be peer-initiated or a force-close by lnd. Atleast that's what I think.

Could you maybe also grep a few logs from above the ones you just posted for this channels outpoint (and perhaps for the channel ID).

Not sure which channel you're referring to.

ellemouton commented 1 year ago

Alright - chatted to the OP offline & we found that the associated channel has been closed but the sweep tx did not broadcast correctly it seems and that is what is showing up in LND.

@WalterSmuts, you are running with a neutrino back-end correct? Would be useful to see some more logs -> my suspicion is that LND gave the tx to neutrino and then neutrino didnt broadcast it properly...

Perhaps would help to turn on debug logs & restart so that the tx is given to neutrino again for broadcast

ellemouton commented 1 year ago

The OP sent me some logs after turning on debug level. Can confirm that:

  1. LND re-sends the tx to neutrino LNWL: Successfully rebroadcast unconfirmed transaction
  2. Neutrino sends an INV with the tx to a bunch of peers BTCN: Sending inv
  3. The peers to ask the OP's node for the tx BTCN: Received getdata
  4. The OP's node sends the tx BTCN: Sending tx

However, the TX is still not appearing in the mempool.

Perhaps the peers are getting the tx and then rejecting it? I have asked the OP to send me the raw hex of the tx so I can try submit it so bitcoind myself to see what error is returned

ellemouton commented 1 year ago

Alrighty! getting closer to the issue:

Tried to submit the tx to the mempool and got: 

error code: -25
error message:
bad-txns-inputs-missingorspent

The output from listchaintxns shows the following:

  "label": "0:sweep",
            "previous_outpoints": [
                {
                    "outpoint": "1223529f41409200453b3452a5db8ff4de0145ac8be222100566d3c209a2c3f4:0",
                    "is_our_output": false
                },
                {
                    "outpoint": "67426e4cbf40a685597450c68f29eb4f7a4d9979eb8eb529a2bb7acdf222153d:0",
                    "is_our_output": true
                }
            ]

The first tx is not a valid tx

ellemouton commented 1 year ago

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

ellemouton commented 1 year ago

related: https://github.com/lightningnetwork/lnd/issues/6241 & https://github.com/lightningnetwork/lnd/pull/6274

WalterSmuts commented 1 year ago

Issue still persisting...

miiiight have to do with the fact that someone else swept the anchor output which then invalidated our own sweep of the anchor: https://blockstream.info/tx/078c3357606d7d959e9e5d3b70409971880c384886511c0b5d333b6604ea521d?input:12

@Roasbeef I believe https://github.com/lightningnetwork/lnd/pull/6274 is supposed to solve this issue. Any idea why it's not working?

robertclarkson commented 1 year ago

I have this issue as well

lncli walletbalance | grep uncon
    "unconfirmed_balance": "6536184",
            "unconfirmed_balance": "6536184"

{
    "utxos": [
        {
            "address_type": 4,
            "address": "bc1pg68tner0yukcra78gtrsj7remuf0vnsdd462gd7fg0u3v2rtnspsmnwsgj",
            "amount_sat": 1471233,
            "pk_script": "5120468eb9e46f272d81f7c742c7097879df12f64e0d6d74a437c943f916286b9c03",
            "outpoint": "f061053dc0dbbc04fc89dd38dd5ea794992e7cab0eb8cd9d27c06add5e5b3429:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1pj3jl7jzg0drqwpqkflemp9cc582vrjkfvd2xepwz7vs2a7tzm40qn85854",
            "amount_sat": 1974652,
            "pk_script": "51209465ff48487b460704164ff3b09718a1d4c1cac963546c85c2f320aef962dd5e",
            "outpoint": "e9e76d85e2eccc466fe70565544de5c4cccbbe1da1400c0b06d4eddbe0d99bde:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1pjtlzgmxj7gx7ncjhd9d945xez6g2nn5v6cfcl6df7zkm9jj6awqsnyp458",
            "amount_sat": 1974505,
            "pk_script": "512092fe246cd2f20de9e257695a5ad0d91690a9ce8cd6138fe9a9f0adb2ca5aeb81",
            "outpoint": "703a0b1dd2a7904edcf84b10b48215ae6895e7c66d21c209028836f1aef72aab:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1pa3jkv94fjg653mjm8hnarzvpca4anveyddey279hjrqumju63jqqz7zf5c",
            "amount_sat": 465929,
            "pk_script": "5120ec656616a9923548ee5b3de7d18981c76bd9b3246b724578b790c1cdcb9a8c80",
            "outpoint": "1443d232358b56c87bcab6a37c2cda6ab5740d1ff50207c2ea10009536a7f648:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1p83v0adgwq3uq2jhwysv6hes47ytzkv7vxqzz0m7agtqplc50m8jqmqsmzg",
            "amount_sat": 649865,
            "pk_script": "51203c58feb50e0478054aee2419abe615f1162b33cc300427efdd42c01fe28fd9e4",
            "outpoint": "261f965ce0c767a11e27f7c224777ae32bffc8879ecf31a7299a1c7ed835de5a:0",
            "confirmations": 0
        }
    ]
}

Getting this error in the logs

2022-11-15 09:55:43.596 [ERR] BTCN: utxo scan failed: couldn't retrieve block 00000000000000000005c735989c24181d50ab693933ccf614c04f5ce0fc6538 from network
2022-11-15 09:55:33.594 [ERR] BTCN: utxo scan failed: couldn't retrieve block 00000000000000000005c735989c24181d50ab693933ccf614c04f5ce0fc6538 from network
robertclarkson commented 1 year ago

using the command 

lncli sendcoins --sweepall --min_confs 0 --addr bc1q....xxx

seems to have collected all of the zero conf transactions into one new transaction

fingers crossed

WalterSmuts commented 1 year ago

Thanks for posting Rob, please let us know what the result is.

robertclarkson commented 1 year ago

Same problem, except now i have 1 zero confirmation transaction stuck and it doesn't seem to have been broadcast by nutrino either!

robertclarkson commented 1 year ago

well the created transaction seems to have a negative amount. that cant be right 

{
            "tx_hash": "2ce2b8d0c014f133eb0cbb5c051c40e5459fabf2b5aae3516d23265d12b13c06",
            "amount": "-6308",
            "num_confirmations": 0,
            "block_hash": "",
            "block_height": 0,
            "time_stamp": "1668584398",
            "total_fees": "6308",
            "dest_addresses": [
                "bc1qte75zsmmdvn4rhppuxqxtaxqh7dlmk886lkcwz"
            ],
            "output_details": [
                {
                    "output_type": "SCRIPT_TYPE_WITNESS_V0_PUBKEY_HASH",
                    "address": "bc1qte75zsmmdvn4rhppuxqxtaxqh7dlmk886lkcwz",
                    "pk_script": "00145e7d41437b6b2751dc21e18065f4c0bf9bfdd8e7",
                    "output_index": "0",
                    "amount": "7208118",
                    "is_our_address": true
                }
            ],
            "raw_tx_hex": "0200000000010729345b5edd6ac0279dcdb80eab7c2e9994a75edd38dd89fc04bcdbc03d0561f0000000000000000000de9bd9e0dbedd4060b0c40a11dbecbccc4e54d546505e76f46ccece2856de7e9000000000000000000ab2af7aef136880209c2216dc6e79568ae1582b4104bf8dc4e90a7d21d0b3a7000000000000000000048f6a736950010eac20702f51f0d74b56ada2c7ca3b6ca7bc8568b3532d243140000000000000000005ade35d87e1c9a29a731cf9e87c8ff2be37a7724c2f7271ea167c7e05c961f26000000000000000000fdb1250ec5b1fed711cb3f2727ca29682f90e6e4811d34aef590fc91765b973e000000000000000000f2f9124b32aeae3709c8f098c74cf2da9294d2ba7af2fef22e25d0a9ee42594100000000000000000001b6fc6d00000000001600145e7d41437b6b2751dc21e18065f4c0bf9bfdd8e7014009822a3c929d74b26a3cd9721d0df093ed3d9196174c40e2cac11410c1a253fb0cc706ae3930980690dca37759e6c800155fbb12e3f6fe2245542d557eda2fd401401cc0411531836c0f99f4e6d0311238582a35b31df4819fec5e45d87f3474f1183a6e2ee54fcdf736556976fb516ad10c6213e24fac9500ed1c578ff867c3fe41014082e65b25aeb916039629ec4856a53a18031e2d8b9a2121695d339530d1cb28efa129dfb3f14b5ac6638c64182af119d4808e3f7c94d1a1e3858334df919a18390140877307ada1a76f413260cd37fbb1f3a00c0698fa9f9fe1e08235f1afc1ceb575cbcee5bd68112825d7d66514541c385eee8d0c2b5034386c0b524dc63b03061801400e556a8ef19fb8dffad81980962b5a27722678521137f4cb6771f9b642f4d89b8d3cccaa80033fc33564a119e9e2a657f2df8a7e9c75560e99841289ba4cdeb10140347dc410e448177c88e339b7de00871042f7440999586676be9b189530177be24f2a111dde9854b80438b1ea4fc6f430dd255eb99b17187eec55952fc8798e990140ec27297f201857112184a88c0644dfd3a29046e747f08dacf4f21b3015a8a29c4251f182ea96cb87ef5bfbc178243d7285b5e7ea6a475d1c4c2707e1cbc1cf4203a60b00",
            "label": "external"
        }
robertclarkson commented 1 year ago

I've tried broadcasting one of the raw transactions from my original message 

02000000000102fd1d8d2fb559d6436a45d12bcdae0b17863188b0bd1a2df046b139e41149a07801000000000000000098383cb259e631f6c19b9ad79ab5d60aeea25bb8d00ef0f823165a99cf4c924f00000000000000000001091c070000000000225120ec656616a9923548ee5b3de7d18981c76bd9b3246b724578b790c1cdcb9a8c8002473044022010b57190fca53a8d89ffd15b1e8beff7302cde78f948aecc34aec50295d96ee902205d8263441ebcd5bb81d0d7a91a556111d31e908daa78de0fb0aed380e905e9c4012821024c821848f3b80302c5625d09ef5b59c1951725cab223d948584f48c70e825f96ac736460b2680140bbffbbabe064fb012f98bed23d7b96b79d728fc59b4df8bba89df08b27b3d3f113ebe932646c1ffafc03d9e545afa4b40d887847ab42488cd1170c62a90f5d10209c0b00

Error is "bad-txns-inputs-missingorspent"

WalterSmuts commented 1 year ago

Got two more stuck transactions:


$ lncli listunspent --unconfirmed_only
{
    "utxos": [
        {
            "address_type": 4,
            "address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
            "amount_sat": 1113186,
            "pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
            "outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1ph0c0c9v3en8k9d003e2rtqlpnpqnuk020rcyep96035we5kfwsuqne96em",
            "amount_sat": 486858,
            "pk_script": "5120bbf0fc1591cccf62b5ef8e543583e198413e59ea78f04c84ba7c68ecd2c97438",
            "outpoint": "f3a7d850701d46ee23affb31d8d9831453323b9be710514e9c28ed5ad20a0d52:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
            "amount_sat": 1051070,
            "pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
            "outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
            "confirmations": 0
        }
    ]
}```
WalterSmuts commented 1 year ago

Added explicit peers about a month ago:

[neutrino]
neutrino.addpeer=btcd-mainnet.lightning.computer
neutrino.addpeer=mainnet1-btcd.zaphq.io
neutrino.addpeer=mainnet2-btcd.zaphq.io
neutrino.addpeer=mainnet3-btcd.zaphq.io
neutrino.addpeer=mainnet4-btcd.zaphq.io

This didn't fix anything but since then I haven't accrued any new unconfirmed tx's:

$ lncli listunspent --unconfirmed_only
{
    "utxos": [
        {
            "address_type": 4,
            "address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
            "amount_sat": 1113186,
            "pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
            "outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1ph0c0c9v3en8k9d003e2rtqlpnpqnuk020rcyep96035we5kfwsuqne96em",
            "amount_sat": 486858,
            "pk_script": "5120bbf0fc1591cccf62b5ef8e543583e198413e59ea78f04c84ba7c68ecd2c97438",
            "outpoint": "f3a7d850701d46ee23affb31d8d9831453323b9be710514e9c28ed5ad20a0d52:0",
            "confirmations": 0
        },
        {
            "address_type": 4,
            "address": "bc1pgptzpw0jr635psy7z4wzm5e74sqlmf96wfqe0uzd9jvpwjjw0adqnek4jc",
            "amount_sat": 1051070,
            "pk_script": "5120405620b9f21ea340c09e155c2dd33eac01fda4ba724197f04d2c98174a4e7f5a",
            "outpoint": "3bc2fa79c945da5f5d457954a2235678ad3e1c1b3a3d7cbd831cff14984cab97:0",
            "confirmations": 0
        }
    ]
}
robertclarkson commented 1 year ago

I should add my issue was solved by running btcd in rescanning mode.

WalterSmuts commented 1 year ago

I ran lnd with --reset-wallet-transactions, waited for LNWL: Finished rescan for. Removed the second two unconfirmed tx, first one still there:

$ lncli listunspent --unconfirmed_only
{
    "utxos": [
        {
            "address_type": 4,
            "address": "bc1p3g7zgu2teq5shu89jflnrzadl4869ft5pl4wl0ppyu43ry07cseqndfkjm",
            "amount_sat": 1113186,
            "pk_script": "51208a3c24714bc8290bf0e5927f318badfd4fa2a5740feaefbc21272b1191fec432",
            "outpoint": "6cb4874ec6bc654fe7f10346d0edb7fee17acbb1fb4df964c8e03d9a4ba97d2b:0",
            "confirmations": 0
        }
    ]
}

Notice new outpoint.

Horndev commented 1 year ago

I seem to be having the same issue. sweeper went nuts with a bunch of txns which never go into mempool until 100% of UTXO was showing as unconfirmed. I did a rescan, now 2 remain.

"utxos": [
                {
                        "address_type": 4,
                        "address": "bc1pfec57mkv3n90xpa3c379rds72ru9ldydkmjanz37kkg9crjxgxpqrvtzpg",
                        "amount_sat": 2076264,
                        "pk_script": "51204e714f6ecc8ccaf307b1c47c51b61e50f85fb48db6e5d98a3eb5905c0e464182",
                        "outpoint": "53085820a65121739be6ffb067aedbfff10ddf7f6883b61e4631b8dc2c5adfad:0",
                        "confirmations": 0
                },
                {
                        "address_type": 4,
                        "address": "bc1py45gpxlexcdvwgne7cmfh8zngd6llqpede02zjwmkdu6fmd8x3vqjd0p72",
                        "amount_sat": 2052362,
                        "pk_script": "51202568809bf9361ac72279f6369b9c534375ff80396e5ea149dbb379a4eda73458",
                        "outpoint": "1e345ec40d3a02ddef1112f9425599e51e72269255f3185effcc05a91261f33a:0",
                        "confirmations": 0
                }
        ]

When I query the transaction from listchaintxns I get the raw tx. When I broadcast, I get no error from lnd, but on external site I get this error:

"Code: -26, Error: insufficient fee, rejecting replacement 53085820a65121739be6ffb067aedbfff10ddf7f6883b61e4631b8dc2c5adfad, less fees than conflicting txs; 0.00000183 < 0.00000194"

Digging deeper, I find that the first input is a lightning anchor, and has been swept already. https://mempool.space/address/bc1qrh433dns65em58nrxc3jmqz80zfjphccawn74xr60feuna8m8wvsdw2j60

It seems that this address is collecting and sweeping lots of lightning anchors. https://mempool.space/address/bc1ptztt50y5xk2pngsf8kfjd5qvek7qg485ks6nj8aq9p2k25mgqrwsy8rsar

So maybe this is not detected by LND, and it's broadcasting a transaction with invalid input? I mean, these are all 330 sat utxos, but this address has collected a ton of these. What's going on?

I am running LND 0.15.4-beta

Horndev commented 1 year ago

likely related to https://github.com/lightningnetwork/lnd/issues/6241

Horndev commented 1 year ago

Ok, the node keeps trying to sweep these already spent utxos, and it's locking up my actual utxos in transactions that will never be confirmed.

M1ch43lV commented 1 year ago

Also likley related to https://github.com/lightningnetwork/lnd/discussions/7420