lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.69k stars 2.09k forks source link

[bug]: can't bump fee of a coop closed channel #8391

Open rkfg opened 9 months ago

rkfg commented 9 months ago

Background

The channel point: https://mempool.space/tx/e49ddf72ead7445430f8e3dae014fa415282fd4c28ebced17d41a7606293abb0#vout=1

It was coop closed by Boltz long ago, right before the ordinals spam began. The tx never confirmed and was purged from my mempool I suppose. Now I try to bump the fee using lncli wallet bumpclosefee --conf_target 6 e49ddf72ead7445430f8e3dae014fa415282fd4c28ebced17d41a7606293abb0:1, however the command exits without any errors or success messages (I expect {} if it succeeds), there are no messages in the log either. I expect this command to use CPFP to bump my tx by spending my output but it doesn't seem to happen.

Your environment

Roasbeef commented 9 months ago

If the fee rate of the transaction you want to bump is below the current mempool exclusion rate, then CPFP may not work. For CPFP to work, the node you're sending the tx to needs to have the parent transaction. It won't have the parent transaction if it was "dropped from the mempool".

There's an upcoming improvement to the protocol that enables the co-op close txn to be RBF'd. This would help your case as the parent transaction is 2 sat/vbyte rn.

rkfg commented 9 months ago

Can't both txs be broadcasted together so that their combined fee rate is above the purge limit? Also, shouldn't there be at least some error reported back from that command? Currently it's impossible to tell whether it worked or not and why.

And what should I do now to recover the funds?

Roasbeef commented 9 months ago

Can't both txs be broadcasted together so that their combined fee rate is above the purge limit?

In theory yes, but in practice no. The Bitcoin protocol doesn't yet support "package relay", which is the feature that would allow both of them to be broadcasted in tandem.

Also, shouldn't there be at least some error reported back from that command

After we merge https://github.com/lightningnetwork/lnd/pull/8345 we'll detect situations like this up front.

But with the way the "mempool" works, we won't know for sure. You only know when something is mined. Even if the first peer accept the txn, all their peers may have a feefiltervalue that causes them not accept the transaction.

And what should I do now to recover the funds?

Unfortunately, all you can do is wait (lower fees to the point where the parent tx is accepted by itself), or try to use some "transaction acceleration" service.

rkfg commented 9 months ago

Thanks for the comprehensive answer! This is truly sad. Does lnd rebroadcast this tx automatically until the tx gets into mempool or should I restart it to do that? I remember some fixes in this area but not sure if they cover this situation.

Roasbeef commented 9 months ago

This is truly sad

Yeah, I wish we had better tools at this point in the venture :/

Does lnd rebroadcast this tx automatically until the tx gets into mempool or should I restart it to do that? I remember some fixes in this area but not sure if they cover this situation.

Yeah lnd will automatically rebroadcast.

One thing I'm working on in the background is a tx accelerator that can operate purely over LN, so keysend to bump a fee for something. Needs some additional entities involved though.

rkfg commented 9 months ago

Good, then I suppose nothing can be done about it now. Thanks for the help!

rkfg commented 9 months ago

So, the channel has finally been closed as the fees dropped. However, the issue was still standing. I tried lncli wallet bumpclosefee --conf_target 1 e49ddf72ead7445430f8e3dae014fa415282fd4c28ebced17d41a7606293abb0:1 but it returned nothing and didn't bump the fee while it should have: the purge cut off at the time was below 2 sat/vB (which is the original fee). I only managed to bump the fee using my own output (CPFP) directly with lncli wallet bumpfee --conf_target 1 --force 49a4185267c68b9c9ebe90b178ba6567f37334a17a7b523a996595217a3ccabb:1. Then a new tx was created and broadcast, however, even then the combined fee was too low, about 10.7 sat/vB while the median fee in the last blocks was around 18 sat/vB so I waited for another day or two until it confirms. This command returned {}. Looks like something isn't right.

guggero commented 9 months ago

The bumpclosefee command is only for force closes. That was made more clear in a recent PR (https://github.com/lightningnetwork/lnd/pull/8350, not yet in a released version). So using lncli wallet bumpfee is correct.

Did you specify a fee rate when using lncli wallet bumpfee? If not, a rate is estimated which might be too low. The response of {} just means "no error happened". See https://github.com/lightningnetwork/lnd/pull/7762

rkfg commented 9 months ago

The bumpclosefee command is only for force closes. That was made more clear in a recent PR (#8350, not yet in a released version).

I suppose it wouldn't be hard to extend it to co-op closes as well? We know our output and we can perform CPFP. Otherwise I need to manually look up the close tx and my output.

Did you specify a fee rate when using lncli wallet bumpfee?

I specified --conf_target 1, I think it's mutually exclusive with --sat_per_vbyte? Also I use feeurl=https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json because of #7756, can't risk it. But I hope those estimates are good enough to calculate the right fee for CPFP.

Roasbeef commented 9 months ago

This PR will allow for natively doing a fee bump (via RBF) on a co-op close channel: https://github.com/lightningnetwork/lnd/pull/8453

Roasbeef commented 9 months ago

I suppose it wouldn't be hard to extend it to co-op closes as well? We know our output and we can perform CPFP. Otherwise I need to manually look up the close tx and my output.

Depends if it fell out of the mempool or not. Architecture wise, it is possible, the first step is that the sweeping module is made aware of the co-op close outputs.

We can morph this issue to track the implementation of that feature.

rkfg commented 9 months ago

We can morph this issue to track the implementation of that feature.

I'm not against that, but I'd still like to know about the fee estimation quirk. Could it have happened because the original coop close tx was purged and rebroadcast later? Or maybe there was some other edge case? I found this in the log when I bumped the fee:

Feb 11 12:22:39 fullnode lnd[420361]: 2024-02-11 12:22:39.743 [ERR] LNWL: Web API does not have a fee rate for target=1, using the fee rate for target=2 instead
Feb 11 12:22:39 fullnode lnd[420361]: 2024-02-11 12:22:39.744 [ERR] LNWL: Web API does not have a fee rate for target=1, using the fee rate for target=2 instead
Feb 11 12:22:39 fullnode lnd[420361]: 2024-02-11 12:22:39.744 [INF] SWPR: Candidate sweep set of size=1 (+0 wallet inputs), has yield=0.04947053 BTC, weight=397
Feb 11 12:22:39 fullnode lnd[420361]: 2024-02-11 12:22:39.745 [INF] SWPR: Sweep candidates at height=829950 with fee_rate=6447 sat/kw, yield 1 distinct txns
Feb 11 12:23:09 fullnode lnd[420361]: 2024-02-11 12:23:09.746 [ERR] LNWL: Web API does not have a fee rate for target=1, using the fee rate for target=2 instead
Feb 11 12:23:09 fullnode lnd[420361]: 2024-02-11 12:23:09.746 [INF] SWPR: Candidate sweep set of size=1 (+0 wallet inputs), has yield=0.04947053 BTC, weight=397
Feb 11 12:23:09 fullnode lnd[420361]: 2024-02-11 12:23:09.772 [INF] SWPR: Creating sweep transaction b843ce7f920a6e8892bd904f7e8bc3b5296a3884ddd9fa3c438d896defbf48ff for 1 inputs (49a4185267c68b9c9ebe90b178ba6567f37334a17a7b523a996595217a3ccabb:1 (TaprootPubKeySpend)) using 6447 sat/kw, tx_weight=445, tx_fee=0.00002868 BTC, parents_count=0, parents_fee=0 BTC, parents_weight=0
Feb 11 12:23:09 fullnode lnd[420361]: 2024-02-11 12:23:09.838 [INF] NTFN: New confirmation subscription: conf_id=10, txid=b843ce7f920a6e8892bd904f7e8bc3b5296a3884ddd9fa3c438d896defbf48ff, num_confs=6 height_hint=829950

It was confirmed almost 16 hours after that, clearly the fee wasn't correct. It's not easy now to reconstruct the mempool structure at that time, but as I wrote the recent blocks had 18 sat/vB median fee and my tx was put more than 6 blocks away from the tip. It doesn't make sense no matter how you look at it if I requested it to confirm in 2 blocks. I understand that it's gonna always be random but my tx should at least be in the second mempool block, not 7-8th. Otherwise it's strictly impossible to confirm in 2 blocks. Either the web API calculates incorrectly, or there's something wrong with the CPFP scenario, maybe an edge case of sorts.

Roasbeef commented 9 months ago

The fee estimation we're hitting is just vanilla bitcoind. Fee estimation is also lagging, if there's a very large spike it may not be 100% reflective of the true block you need to get into. Strike has a composite API that they offer that combines other fee sources.

Otherwise it's strictly impossible to confirm in 2 blocks.

Confirming in 2 block basically requires bidding at the very top of the mempool. This isn't something that you always want to do. We'll also take into account exactly how much we'll receive on the other side as well, to ensure we don't spend all the funds to fees.

rkfg commented 9 months ago

The fee estimation we're hitting is just vanilla bitcoind. Fee estimation is also lagging, if there's a very large spike it may not be 100% reflective of the true block you need to get into.

I don't see any spikes in that time frame, it was pretty smooth between 18 and 20 sat/vB.

Confirming in 2 block basically requires bidding at the very top of the mempool. This isn't something that you always want to do.

Weird, does it mean that the option --conf_target simply doesn't work? Because I specified that explicitly and even used --force, so I suppose it should mean "yes, do as I say".

We'll also take into account exactly how much we'll receive on the other side as well, to ensure we don't spend all the funds to fees.

The yield is 4947053 sats, I don't think this applies here. The fee was set to 2868 sats and the effective fee rate was 10 sat/vB. If it were 5200 sats, the effective rate would have been 18.4, and at the time that'd have made sure it's in the first block (unless it gets pushed back before it's mined but we can't do anything about it). I'm not convinced the current logic is correct and/or documented properly.

Roasbeef commented 8 months ago

Weird, does it mean that the option --conf_target simply doesn't work? Because I specified that explicitly and even used --force, so I suppose it should mean "yes, do as I say".

Not that it doesn't work, but that "getting into the next two blocks" is always a moving target. If you bid 10 sat/byte, but the min is at 50 sat/byte, bumping is needed to get the fee to that point.

The PR I linked above, and #8148 (the entire series) will resolve this (dynamic fee bumping, RBF for coop close).