Unable to open channel because "mempool min fee not met", channel still shows as pending.

jpentland commented 6 years ago

Background

Tried to open open a channel but it failed due to "mempool min fee not met". Channel still shows as pending.

Your environment

version of lnd : commit e0baa4969035fc555b185d21acaa0b09aecdef6f (mainnet)
which operating system (uname -a on *Nix): Linux bitcoin-node 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux (raspberry pi)
version of btcd, bitcoind: Bitcoin Core Daemon version v0.16.1

wallet balance before:

lncli walletbalance
{
"total_balance": "115918",
"confirmed_balance": "115918",
"unconfirmed_balance": "0"
}

bitcoind settings:

rpcuser=bitcoinrpc
rpcpassword=<redacted>
dbcache=100
maxorphantx=10
maxmempool=5
minrelaytxfee=0.00000000
server=1
listen=1
txindex=1
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28332

Note that minrelaytxfee was set to 0 to ensure that low TXs created by LND get forwarded.

Steps to reproduce

Executed openchannel to LIGHTNING.SCOT node:

lncli openchannel --node_key 02a456b816697c6b8407ee2176475d74b790a96143bc30e896e8bf7a8852a8b637 --local_amt 100000 --push_amt 0

Expected behaviour

Channel opens, or fails with reason given.

Actual behaviour

lncli blocked and did not return from the RPC call, had to be killed with CTRL+C.

Channel did not open due to "mempool min fee not met".

Channel stuck in pending state, although TX was not broadcast anywhere.

Relevant log output:

2018-07-25 13:03:05.506 [INF] LNWL: Inserting unconfirmed transaction 2ae81716cc429e7ce0441469f2f173068d4a0888f669e4fbb67821100e362175
2018-07-25 13:03:05.601 [ERR] FNDG: Unable to complete reservation sign complete: -26: 66: mempool min fee not met
2018-07-25 13:03:05.602 [DBG] FNDG: Failing funding flow for pendingID=900048e234a9c4653880745b6b86c83523bd9c4709b2f6de8a8dd6b275fc3bd2:
-26: 66: mempool min fee not met
2018-07-25 13:03:05.602 [INF] FNDG: Cancelling funding reservation for node_key=02a456b816697c6b8407ee2176475d74b790a96143bc30e896e8bf7a
8852a8b637, chan_id=900048e234a9c4653880745b6b86c83523bd9c4709b2f6de8a8dd6b275fc3bd2
2018-07-25 13:03:05.602 [ERR] FNDG: unable to cancel reservation: unable to cancel reservation: attempted to cancel non-existent funding
state
2018-07-25 13:03:05.602 [DBG] FNDG: Sending funding error to peer (02a456b816697c6b8407ee2176475d74b790a96143bc30e896e8bf7a8852a8b637):
(*lnwire.Error)(0x13a3bbc0)({
ChanID: (lnwire.ChannelID) (len=32 cap=32) 900048e234a9c4653880745b6b86c83523bd9c4709b2f6de8a8dd6b275fc3bd2,
Data: (lnwire.ErrorData) (len=36 cap=48) {
00000000  66 75 6e 64 69 6e 67 20  66 61 69 6c 65 64 20 64  |funding failed d|
00000010  75 65 20 74 6f 20 69 6e  74 65 72 6e 61 6c 20 65  |ue to internal e|
00000020  72 72 6f 72                                       |rror|
}
})
2018-07-25 13:03:05.603 [DBG] PEER: Sending Error(chan_id=900048e234a9c4653880745b6b86c83523bd9c4709b2f6de8a8dd6b275fc3bd2, err=funding
failed due to internal error) to 206.189.124.227:9735

Output from lncli afterwards:

lncli pendingchannels
{
"total_limbo_balance": "164076",
"pending_open_channels": [
{
"channel": {
"remote_node_pub": "02a456b816697c6b8407ee2176475d74b790a96143bc30e896e8bf7a8852a8b637",
"channel_point": "2ae81716cc429e7ce0441469f2f173068d4a0888f669e4fbb67821100e362175:1",
"capacity": "100000",
"local_balance": "99638",
"remote_balance": "0"
},
"confirmation_height": 0,
"commit_fee": "362",
"commit_weight": "600",
"fee_per_kw": "500"
}
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
... removed since it's unrelated ...
],
"waiting_close_channels": [
]
}

Roasbeef commented 6 years ago

You'll need to lower the min relay fee for your node.

jpentland commented 6 years ago

The min fee was set to 0 in order to make sure that wasn't the issue. I'm also digging through this myself slowly but the code is new to me.

halseth commented 6 years ago

I'm not sure why it returns a "mempool min fee not met" error, but the error is definitely not handled they way it should.

The funding tx is broadcasted from within the wallet (https://sourcegraph.com/github.com/lightningnetwork/lnd@e0baa49/-/blob/lnwallet/wallet.go#L1093) after the reservation context is already cancelled (https://sourcegraph.com/github.com/lightningnetwork/lnd@e0baa49/-/blob/lnwallet/wallet.go#L1054). This leads to the funding manager trying to fail the funding flow, but since the channel already is transitioned to the openChannelBucket in the DB, it won't be cancelled.

I think the first step to fixing this would be to move the responsibility of broadcasting the funding tx to the fundingmanager. That way we can handle the broadcasting error separately from the other errors returned from the wallet.

Handling the broadcasting error would be the second step. We could

1) keep the pending channel around, and retry broadcasting the funding transaction at a later point (we currently do this at startup), and timeout/double spend the transaction later if it doesn't eventually confirm (see #386). 2) assume that a funding tx that fails to broadcast will never make it into the chain, and just cancel the funding flow, deleting the channel from the openChannelBucket.

I think 2) could be a bit dangerous, as there's no real guarantee that the tx won't eventually confirm (it might reach the network do to some bug say), and then we won't be able to reclaim the funds without some added complexity (keeping the commitment state around until we are sure the inputs are double spent).

Pursuing 1) in tandem with #386 (and maybe #1184) seems to be a safer approach, as this is something we would need to handle regardless. If the mempool min fee is not met, it can be broadcast together with a CPFP to increase the fee.

halseth commented 6 years ago

https://github.com/lightningnetwork/lnd/pull/1635 should handle the fist step mentioned in the above comment. Still won't handle that the transaction never gets published.

halseth commented 6 years ago

@wpaulino has pointed out that the "min mempool fee" error likely is a result of the maxmempool=5 configuration, making the bitcoind node keep only the 5MB of highest paying transactions in the mempool.

I think it is strange if bitcoind doesn't prioritize local transactions in the mempool, but I haven't looked into this.

We should also look into why the fee estimator doesn't return a fee that would ensure the transaction we create gets accepted into the mempool.

halseth commented 6 years ago

Also making a note that we should revisit some of the errors returned from PublishTransaction, as they don't encapsulate all the errors we might want to handle: https://github.com/lightningnetwork/lnd/blob/92b0b10dc75de87be3a9f895c8dfc5a84a2aec7a/lnwallet/btcwallet/btcwallet.go#L424

alexbosworth commented 6 years ago

I also have this issue, stuck funds due to a channel open that used a fee below what I instructed it to open with.

Given that it's a funding transaction and not jointly controlled - wouldn't RBF be the strategy over CPFP?

halseth commented 6 years ago

Problem with RBF is that it changes the funding outpoint, rendering the signed commitment invalid. If RBF is used a new channel must be created with the replacement, then the old channel can be canceled.

alexbosworth commented 6 years ago

I guess it could also require changing the funding amount if you gave more over to fees, so definitely an interactive process unless you are cancelling the channel and spending back to yourself. I'm not sure CPFP will work in a situation like mine though because if the transaction fee is too low the parent fails to relay and CPFP isn't effective

alexbosworth commented 6 years ago

Also I should mention that I didn't see this tracked specifically but the fee estimates targeting 1 sat/byte can result in off-by-one results which are 1 satoshi short of 1 sat/byte which causes stuck tx.

lightningnetwork / lnd