double spend unconfirmed and purged channel open?

[sendbitcoin]

A couple weeks ago (months?) I opened 4 channels at 3 sats/byte. I guess I low balled it way to hard as they never got confirmed. Over the past couple weeks the mempool really filled up and these Transactions were purged from the mempool.

I can still see them under pending channels and have a bunch of sats stuck (I think?). The problem is that these transactions have been purged from the mempool and even when I restart LND it seems they do not get rebroadcast. (I checked with my own node with getmempoolentry)

When i start lnd i can see these entries in the log for each pending channel:

2021-05-04 20:41:48.055 [ERR] FNDG: Unable to advance pending state of ChannelPoint(ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d:1): error waiting for funding confirmation for ChannelPoint(ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d:1): waiting for fundingconfirmation failed

2021-05-04 20:43:05.641 [INF] NTFN: Historical spend dispatch finished for request outpoint=ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d:1, script=0 XXXXXXXXXXXa4 (start=681931 end=681931) with details: <nil>

2021-05-04 20:43:12.994 [INF] LNWL: Inserting unconfirmed transaction ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d
2021-05-04 20:43:13.003 [INF] LNWL: Removed invalid transaction: (*wire.MsgTx)(0xc003447240)({

XXX

})

2021-05-04 20:43:13.004 [ERR] FNDG: Unable to rebroadcast funding tx 02000000000102ed2a1bb1a21a97d60ee214af081099a7ab4e8db20f8a0c2c913aaf1b8e85717700000000000000000071d1366bcd7eab7b173a7910f7b1912c940c25ab1377a7ede92cd962806061ed000000000000000000029b1b000000000000160014b2195303b6f578e26fef019a38fde16e1256731c2a2c0a0000000000220020f1cb9ea49ea3e91ad892dff466e042922ac2e67b9476dc588e2ac75520d6c1a402473044022079d1542929bdd38cdd76013153d5f2f98c32a9ed14d3bc0594c3e63d64a25c4d0220509a9fc7a8188387b59fa63b7b4c7f677b3e5196e05ab0481a4a71d2b313bcb10121033ab92a0432314740d5b72f308ae706c9c44c5b0a9ea09fdc15c8c4fdf152e58002483045022100ad6f8413bb816480c94acfde7f628dba9a9babd7012f4f31b3299bf5ac3034820220373abd824df399f850fc56762b208b24a370e3a177c49325e48dbdcdc2a41670012103f333c42df612882427a4b7486a594fd13f51bec0c9597b9dbb0dd9fdd5923ce300000000 for ChannelPoint(ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d:1): transaction rejected: output already spent
2021-05-04 20:43:13.005 [INF] NTFN: New confirmation subscription: conf_id=1, txid=ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d, num_confs=3 height_hint=681931
2021-05-04 20:43:13.005 [INF] FNDG: Waiting for funding tx (ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d) to reach 3 confirmations

Seeing as it says "output already spent" I'm a bit confused now... Are these channel opens actually stuck and tying up Sats or have the inputs used for those channel opens long since been spent? (It is pretty likely that I used an unconfirmed input to create these channel open txs, as i did that quite often in case that matters)

I would like to doublespend the inputs used for those channel opens to get my sats back... problem is that i do not know the inputs used for these unconfirmed txs... and according to @alexbosworth psbt wouldn't except inputs that are already being spent anyways. I tried to build a PSBT and specify the inputs using the unconfirmed outputs (which i now realize makes no sense) but i always get this error:

[lncli] error parsing outputs JSON: invalid character '\'' looking for beginning of value

which is weird since the command i'm running doesnt even contain an \ ^^ and i'm basically copy pasting from https://docs.lightning.engineering/lightning-network-tools/lnd/psbt

Couple questions:

Are these channel opens actually stuck and tying up sats or does lnd just THINK that they are but actually the utxo have been spent already?

If lnd just thinks they are stuck how do i get it to forget about these pending channels (do i just abandonchannel or what?) note added: abandonchannel is a dangerous command and can lead to loss of funds if used incorrectly - ask in the LND slack channel if unsure

If these Txs are actually stuck and tying up funds, how do i free them:

How do i rebroadcast these TXs so I can bump the fee and then force close them? (and is this advisable as it's been more then 2 weeks and my counterparty has long since forgotten about these channels)

How do i find the Inputs that were used for these unconfirmed TXs so that i might doublespend them and get out of this mess (and how do i do that using psbt, i always get the above error). lncli lstchaintxns results in a file with 18k line so i wouldnt even know where to start looking ^^

I'm open to suggestions as to how to get these sats back in my control... or y'know... make lnd stop showing them as pending...

[wpaulino]

Are these channel opens actually stuck and tying up sats or does lnd just THINK that they are but actually the utxo have been spent already? 2021-05-04 20:43:13.004 [ERR] FNDG: Unable to rebroadcast funding tx 02000000000102ed2a1bb1a21a97d60ee214af081099a7ab4e8db20f8a0c2c913aaf1b8e85717700000000000000000071d1366bcd7eab7b173a7910f7b1912c940c25ab1377a7ede92cd962806061ed000000000000000000029b1b000000000000160014b2195303b6f578e26fef019a38fde16e1256731c2a2c0a0000000000220020f1cb9ea49ea3e91ad892dff466e042922ac2e67b9476dc588e2ac75520d6c1a402473044022079d1542929bdd38cdd76013153d5f2f98c32a9ed14d3bc0594c3e63d64a25c4d0220509a9fc7a8188387b59fa63b7b4c7f677b3e5196e05ab0481a4a71d2b313bcb10121033ab92a0432314740d5b72f308ae706c9c44c5b0a9ea09fdc15c8c4fdf152e58002483045022100ad6f8413bb816480c94acfde7f628dba9a9babd7012f4f31b3299bf5ac3034820220373abd824df399f850fc56762b208b24a370e3a177c49325e48dbdcdc2a41670012103f333c42df612882427a4b7486a594fd13f51bec0c9597b9dbb0dd9fdd5923ce300000000 for ChannelPoint(ddf10de0928cc578234e3e4c867fbb6cb7009dfcfed6728333ad93a0221c991d:1): transaction rejected: output already spent

It seems this is the case based on your rebroadcast attempts always failing with output already spent. You can decode that transaction to inspect its inputs: 7771858e1baf3a912c0c8a0fb28d4eaba7991008af14e20ed6971aa2b11b2aed:0 and ed61608062d92ce9eda77713ab250c942c91b1f710793a177bab7ecd6b36d171:0. The first input doesn't seem to exist in the chain, and the second has already been spent.

If lnd just thinks they are stuck how do i get it to forget about these pending channels (do i just abandonchannel or what?)

Yes, running abandonchannel should remove them. If all of the other channels fail to broadcast with the same error as the one above, it should be safe to remove them all.

[sendbitcoin]

Hmm just to make sure i checked all of them and i'm a bit worried as I still don't fully understand wtf is going on ^^

for two of them i can understand where the money went since i can follow the inputs after decoding the tx (thanks for the tip)

But for these two I don't understand at all what's happened:

02000000000102924a59a14cac76e2db3cc4a750cd527b3ce504c62451609f95bba7946ad01d9d000000000000000000911ab5bdf5b70b6b2378af517acede479cdf6c90af96bac98f4fd06af44cc8cc00000000000000000002a1f8070000000000160014f9948141077b7a7bb8e58c57851cdb95aece58652a2c0a00000000002200208e36ad8830cea7cfa78e7ef12eb6ce7b8b53280c9378232a87ca13ed21e595b502483045022100a15b01859cdbd5c10e199a41c7a1b42237721d081c21592954cba0f186220c1f02207226f758783a18de6afc08e4ddef284e42f1fa37a96432cd6fe6932357c688ca0121031110615934dcb396407c40ec3bb171048dd7f718c8705067e579b5270d73f7db02483045022100bd0c351f720c2b743d6b3a7c3ea6d28a28165ce05ee3bec4d7ad0a82c04c839d02202682c53b91b66e509a41a1d3d2ab6b49a8395f9d5a7bdcb44489bcbe44a28d22012103db7ae568a233ab84f3613e017a449a67176d983b87a68a055958c486d80ae17a00000000

{ "addresses": [ "bc1q3cm2mzpse6nulfuw0mcjadkw0w94x2qvjduzx258egf76g09jk6su0zvqx", "bc1qlx2gzsg80da8hw8933tc28xmjkhvukr9z46rk2" ], "block_height": -1, "block_index": -1, "confirmations": 0, "double_spend": false, "fees": 0, "hash": "c34698144914cff0337e26f816b3b08c60647842f8602cf15165fe45fa6a560f", "inputs": [ { "age": 0, "output_index": 0, "prev_hash": "9d1dd06a94a7bb959f605124c604e53c7b52cd50a7c43cdbe276ac4ca1594a92", "script_type": "empty", "sequence": 0 }, { "age": 0, "output_index": 0, "prev_hash": "ccc84cf46ad04f8fc9ba96af906cdf9c47dece7a51af78236b0bb7f5bdb51a91", "script_type": "empty", "sequence": 0 } ], "opt_in_rbf": true, "outputs": [ { "addresses": [ "bc1qlx2gzsg80da8hw8933tc28xmjkhvukr9z46rk2" ], "script": "0014f9948141077b7a7bb8e58c57851cdb95aece5865", "script_type": "pay-to-witness-pubkey-hash", "value": 522401 }, { "addresses": [ "bc1q3cm2mzpse6nulfuw0mcjadkw0w94x2qvjduzx258egf76g09jk6su0zvqx" ], "script": "00208e36ad8830cea7cfa78e7ef12eb6ce7b8b53280c9378232a87ca13ed21e595b5", "script_type": "pay-to-witness-script-hash", "value": 666666 } ], "preference": "low", "received": "2021-05-04T23:58:40.893410121Z", "relayed_by": "35.171.3.160", "size": 384, "total": 1189067, "ver": 2, "vin_sz": 2, "vout_sz": 2, "vsize": 221 }

Neither of the inputs can be found:

9d1dd06a94a7bb959f605124c604e53c7b52cd50a7c43cdbe276ac4ca1594a92 ccc84cf46ad04f8fc9ba96af906cdf9c47dece7a51af78236b0bb7f5bdb51a91

How is that even possible?

020000000001016805d5dfe5cf8ee5b41eed0e589b06cee19ad8959a6c8a491b9f6742a836b462000000000000000000025591010000000000160014fe54543f9ac5478329e2aa96284876616fdb19ed2a2c0a0000000000220020f2a8f3c17b8bd13bc4ef3a611213323fac1b9d5e6d6437fc81d38a409ba64a050247304402206a2b9f0a01e688e4e7a036fd50a740e01e51312f4744f7790af0cb7dbd5cd1d802205d590a33cadf91f8557c3011d4895f61faee4c7edf24fa0cab228586839fdc83012103bbac5ff9f1f28f3aa267f41b61a0f8187e91c09142472b6938068312a4db591600000000

{ "addresses": [ "bc1qle29g0u6c4rcx20z42tzsjrkv9hakx0dhzr6uy", "bc1q72508stm30gnh3808fs3yyej87kph827d4jr0lyp6w9ypxaxfgzscw0kju" ], "block_height": -1, "block_index": -1, "confirmations": 0, "double_spend": false, "fees": 0, "hash": "cbdfb150607d95c8076467a856a99feaa072c8c6183b83fcf43a16dd790d6436", "inputs": [ { "age": 0, "output_index": 0, "prev_hash": "62b436a842679f1b498a6c9a95d89ae1ce069b580eed1eb4e58ecfe5dfd50568", "script_type": "empty", "sequence": 0 } ], "opt_in_rbf": true, "outputs": [ { "addresses": [ "bc1qle29g0u6c4rcx20z42tzsjrkv9hakx0dhzr6uy" ], "script": "0014fe54543f9ac5478329e2aa96284876616fdb19ed", "script_type": "pay-to-witness-pubkey-hash", "value": 102741 }, { "addresses": [ "bc1q72508stm30gnh3808fs3yyej87kph827d4jr0lyp6w9ypxaxfgzscw0kju" ], "script": "0020f2a8f3c17b8bd13bc4ef3a611213323fac1b9d5e6d6437fc81d38a409ba64a05", "script_type": "pay-to-witness-script-hash", "value": 666666 } ], "preference": "low", "received": "2021-05-05T00:41:27.926003074Z", "relayed_by": "34.224.78.227", "size": 234, "total": 769407, "ver": 2, "vin_sz": 1, "vout_sz": 2, "vsize": 153 }

this input can also not be found:

62b436a842679f1b498a6c9a95d89ae1ce069b580eed1eb4e58ecfe5dfd50568

Any idea what's going on here so i might better understand? like how can an input just vanish like that... i mean at one point these were valid channel opening txs, surely the inputs must be real and exist... I'm confused how they could just not exist on the chain?

[wpaulino]

We've seen instances of ghost inputs before due to wallet bugs (no known cases recently) or using unconfirmed inputs which were double spent and never confirmed. Were the channels funded with unconfirmed inputs? Do the txids of the missing inputs correspond to the other stuck pending channels you have? Can you find any of these txids in your logs?

[Gridflare]

Hi I'm having a similar issue, but with further complications.

I cannot open channels at all right now without using psbt and an external wallet. The last one I opened from lnd also got stuck with output already spent. I also cannot sweep the lnd wallet, with a similar error.

My best guess is that my previous attempt at CPFP fee bumping got screwed up. 2 of 4 channels I tried to bump got through, the other 2 are still stuck.

I tried using chantools to delete the channels but got a database version error

$ lnd --version
lnd version 0.12.99-beta commit=

I've extracted the inputs from the logs, there are a lot of them, 17 in total. Should I post them here, or can you help me understand how to look them up?

I'm also surprised that the change outputs from these don't show up in unconfirmed_balance. Running openchannel certainly consumed balance even though the txn was rejected.

I also have a stuck waiting_close_channel, but I believe that to be an unrelated issue.

[Gridflare]

I wrote a quick script to run the inputs past blockchain.info and check the "spent" attribute. The results look a bit scary to me

import requests

txids = []
with open('stuckinputs.txt') as f:
    for line in f:
        if line == '\n': continue
        txid, point = line.split(':')
        txids.append((txid, point.strip()))

for txid, point in txids:
    r = requests.get(f'https://blockchain.info/rawtx/{txid}')
    tx_data = r.json()

    print(txid, point,
          r.status_code if 'out' not in tx_data else ', '.join(
            [str(t['spent']) for t in tx_data['out']])
          )

The resulting output for my node (newlines added to delimit which channel they belong to)

3cb3e151444cc150d7ced0af0143ac6ec7623285aa8eb83b0c534a5f0762cae3 1 True, True
437740973c8bfacf67e44dd317d3012164f8fd99e34ef47a05edc085a8306c6b 0 True, True
56883b4e1c1e857811659268bc5ad599463f2354805e03c51f907be073bef526 0 True, True
7231bd261b7cf57bf5410767137fc77d4dfba7523631432a70ab194d8aac5bff 0 False, False
7938360cba26645578f09801238527de8e1bf3bafab490d4aed7cc9b220f083f 0 False
97f941a56f524f90355ad06a391e1cf0e63c68901cb3dec0d7e9ad62c9863b2c 0 False, True
c411f6dee5bbd89d0127ad5c09db68a8aec0e572b9eddfb3870ba21803002f55 0 False, False
e89ff1692eb685f286167b85d952b60b6c0809df894f2cd3c9c0ed5b8904b462 0 True, True

3cb3e151444cc150d7ced0af0143ac6ec7623285aa8eb83b0c534a5f0762cae3 1 True, True
37740973c8bfacf67e44dd317d3012164f8fd99e34ef47a05edc085a8306c6b 0 404
56883b4e1c1e857811659268bc5ad599463f2354805e03c51f907be073bef526 0 True, True
7938360cba26645578f09801238527de8e1bf3bafab490d4aed7cc9b220f083f 0 False
97f941a56f524f90355ad06a391e1cf0e63c68901cb3dec0d7e9ad62c9863b2c 0 False, True
b25584ed81f2b4e59988e5e2dedbfb9751b56976e0d397bb5e820365e94161df 0 True
c411f6dee5bbd89d0127ad5c09db68a8aec0e572b9eddfb3870ba21803002f55 0 False, False
89ff1692eb685f286167b85d952b60b6c0809df894f2cd3c9c0ed5b8904b462 0 404

8e65d7698faf3b55c7244b9bf89b7c09e133f215140a517ebb619d6ac81b4e2f 1 False, True

[wpaulino]

The last one I opened from lnd also got stuck with output already spent.

Were unconfirmed inputs used for any of your open channel attempts?

I tried using chantools to delete the channels but got a database version error

note added: abandonchannel is a dangerous command and can lead to loss of funds if used incorrectly - ask in the LND slack channel if unsure

You should instead use abandonchannel available in a debug build of lnd which you can obtain by just running make. You should only do this for channels which you know will not confirm due to having an invalid input.

37740973c8bfacf67e44dd317d3012164f8fd99e34ef47a05edc085a8306c6b 0 404 89ff1692eb685f286167b85d952b60b6c0809df894f2cd3c9c0ed5b8904b462 0 404

So it seems these were the only invalid inputs? Are these still being returned as part of listunspent? Are you able to tell where they originated from based on your logs (searching for all instances of the txid should tell you how)?

[Gridflare]

Were unconfirmed inputs used for any of your open channel attempts?

Don't know for sure, but I don't usually open channels when I have a pending balance, and I currently have no reported pending balance. These were all done with lncli openchannel.

Are these still being returned as part of listunspent?

Turns out these were copy and paste errors

437740973c8bfacf67e44dd317d3012164f8fd99e34ef47a05edc085a8306c6b 0 True, True
e89ff1692eb685f286167b85d952b60b6c0809df894f2cd3c9c0ed5b8904b462 0 True, True

All of these stuck channels have at least one input that blockchain.info reports as spent.

Searching my logs for all of these inputs would be nuts, some are months old.

If I abandonchannel the stuck balance will reappear in my wallet, correct?

Then I still have to get rid of the bad UTXOs somehow, on my hardware reset-wallet-transactions takes two days, an alternative would be appreciated.

[wpaulino]

If I abandonchannel the stuck balance will reappear in my wallet, correct?

Nope, that'll just get rid of the pending channel. You'll need to force a rescan to get rid of the bad UTXOs so that you can continue opening channels using lnd's wallet.

Searching my logs for all of these inputs would be nuts, some are months old.

Since you're still getting output already spent errors when opening channels with lnd's wallet, we should be able to easily identify the offending input(s) by cross-checking the spentness of the inputs returned from listunspent with a bitcoin node/explorer. It could take a while if you have a lot of UTXOs, but seems worthwhile as it could potentially help us spot a bug.

[Gridflare]

I figured I could script this too, but I'm having a lot of trouble compiling and importing walletrpc, I might open a separate issue for that.

Manually checking, I find that 4 of my 11 UTXOs have zero balance according to a block explorer.

[wpaulino]

You don't need walletrpc to access listunspent. If you want to access walletrpc though, you need to compile with make tags=walletrpc.

Manually checking, I find that 4 of my 11 UTXOs have zero balance according to a block explorer.

Are you able to deduce where any of those 4 invalid UTXOs came from with your logs?

[Gridflare]

You don't need walletrpc to access listunspent

To interface with lnd over gRPC I need it, ListUnspent with LightningStub is deprecated.

Are you able to deduce where any of those 4 invalid UTXOs came from with your logs?

By checking block explorers:

First one is change from a channel open. The channel never routed anything and might have been one of my CPFP targets. The second one is the same situation.

Third one I'm unsure where it came from (exchange withdrawal?), but it was spent in the same txn as the fourth.

Fourth one is the return from a channel I closed 3 months ago, making it about a month older than the others. I don't believe I was having any issues at the time. It was successfully used to open a channel to 1ML that has since been closed.

Turns out all 4 were spent in the same transaction https://mempool.space/tx/97f941a56f524f90355ad06a391e1cf0e63c68901cb3dec0d7e9ad62c9863b2c

More for my reference than yours I think, the output of the closed channel to 1ML was then sent to the Electrum wallet I use for batch funding.

[wpaulino]

Turns out all 4 were spent in the same transaction https://mempool.space/tx/97f941a56f524f90355ad06a391e1cf0e63c68901cb3dec0d7e9ad62c9863b2c

Interesting, so somehow those inputs weren't marked as spent after the transaction confirmed? Unfortunately we don't have much logging in that area to tell. Do you see Marking unconfirmed transaction 97f941a56f524f90355ad06a391e1cf0e63c68901cb3dec0d7e9ad62c9863b2c in your logs?

[Gridflare]

Unfortunately that was confirmed in early March, the rotated logs from lnd only go back to early April. Such a message does not exist in my recent logs either.

[Gridflare]

The plot thickens. My node was offline when that transaction confirmed, I was performing a rescan due to issues that are probably related.

The reason for the rescan was stuck pending open channels. At the time I assumed it was a fee issue, and had not checked the startup logs to see the broadcast error.

[wpaulino]

My node was offline when that transaction confirmed, I was performing a rescan due to issues that are probably related.

As far as I'm aware this shouldn't have an effect on those outputs not being marked as spent, but without logs we can't really tell for sure.

[Roasbeef]

Closing due to inactivity.