recovery/btcwallet: detect funds in tweakless/static_remote output addresses on wallet recovery

statusquont commented 3 years ago

Background

I had LND via MyNodeBTC with 3 open channels. The RaspberryPi device with myNode got stolen. I have the seed and a recent channel.backup file. I also have a recent screenshot (RTL) of with channel peer names and the balances.

I downloaded a new copy of myNode for VMWare, resynced the blockchain, restored the LND wallet with seed and channel.backup file. Immediately after restoring from backup, one of the 3 channels (with LNBig https://1ml.com/channel/666722960468934656) broadcast a Remote Force Close transaction (08602f9722dc56cc7ecd5c1b5bd9a4faa67fa8b9e152efcd595ee4d30e8fea0f). All those sats (minus fees) came back to the LND hot wallet as expected.

The 2 other channels did not.

For a few days there were 14 peers (some quite old) listed within RTL under Connections > Channels > Pending > Waiting Close. Those have all since moved over to Connections > Channels > Closed. There are no other channels in any other section of RTL.

The sats from the 2 "stuck" channels have not been returned to the wallet yet. I had reached out to those peers to ask them to close the channel from their side (since I couldn't see any way to do it from my end) which they did. According to those peers the closing tx's are as follows:

Channel: https://1ml.com/channel/698115116873875457 TXID: 2c9d90f23f458f44530514835428f7bacd280ec99e9d2583e2e422b2b23ae7db Previous outbound liquidity channel balance: 933,626 sats

Channel: https://1ml.com/channel/687950131854508032 TXID: 5ce8c15322f82bfc3f180a13afe2f58ab222e095e533f60068498d943456aea2 Previous outbound liquidity channel balance: 765,789 sats

Since one of the channels was with Bitfinex, I reached out to their support team for additional help. According to them the 933,626 sats were supposed to be released as of block 657,312. I'm not entirely sure why that was the block height they provided (exactly 600 after tx block?), but regardless, the sats are not showing up in the wallet, either RTL or the CLI and that block has since past.

Your environment

version of lnd - v0.11.0-beta
which operating system (uname -a on *Nix) - Debian/RaspOS via VM VirtualBox
version of btcd, bitcoind, or other backend - 0.20.1
any other relevant environment details

Expected behaviour

It would be great if there was a command available to figure out where all of a users sats are. Be it in active channels, the hot wallet, or in limbo on-chain pending some block/time.

Actual behaviour

I currently have no idea if/when the 1,699,415 will ever return to the wallet. It's not a life-changing amount, but since I am dealing with a scenario where the node is no longer in my possession, it would be great to know how much longer I should be expecting to wait so I could decide to move funds now, or wait X more hours to sweep all.

Any help here in determining if/when the funds will actually return would be greatly appreciated! And if I left out any important details, please do let me know.

guggero commented 3 years ago

Needing to recover a node is never a nice thing. And it's a rare enough thing to happen that most UIs like RTL aren't best suited to show all the necessary information. Additionally, recovery of the channels fully relies on the other nodes to cooperate and not all implementations behave the same way.

I'm pretty confident you'll get the sats back. To assist you, can you please provide me the following information (feel free to DM me on Slack or Keybase):

The output of lncli pendingchannels
The full lnd.log grepped by 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f
The full lnd.log grepped by 033d8656219478701227199cbd6f670335c8d408a92ae88b962c49d4dc0e83e025

I assume you only have the channel.backup file from the stolen node and no other files that were backed up at one point?

guggero commented 3 years ago

Update: Talked to @statusquont offline, we were able to recover all funds.

The main issue was an old channel backup that did not contain the two mentioned channels. Since both of them were opened with tweakless/static_remote commitment outputs, they should have been picked up by lnd. On checking the code, I noticed that this still isn't the case and started working on a fix for this.

Changing the title since another command in lncli wouldn't have fixed this, lnd just didn't know about the tweakless output addresses.

thunderbiscuit commented 2 years ago

Hi there. I have a similar issue, although with recovery of older channels. This issue was pointed out to me on the Slack server, and I'm wondering if I could DM someone to confirm my understanding that indeed funds will not be recoverable.

guggero commented 2 years ago

If the channel was created with lnd v0.6.0-beta or earlier then you likely need a secret from the remote peer to recover the funds. That version was released early 2019. If the channel was created later, it should be possible to recover. Try chantools sweepremoteclosed (download here) with a large enough recovery window. If that finds something, cool. If not, you're likely out of luck.

thunderbiscuit commented 2 years ago

I do not have any backup left of the old node unfortunately. Looking at the readme for the chantools repo, I see Step 4 mentions that

The recovery can only be continued if you have access to some version of the crashed node's channel.db

If not, you must go to step 11. I think step 11 requires you have knowledge of the peer with whom your channel was (if I understand correctly). I do not have this information either.

Does it mean that the chantools set of tools cannot help me?

guggero commented 2 years ago

Those instructions in the main README are a kind of "general recovery walkthrough" to tell you what commands to use. But the specific command I mentioned doesn't require any input other than the seed.

Roasbeef commented 1 year ago

Closing as a duplicate of https://github.com/lightningnetwork/lnd/issues/4778

starius commented 11 months ago

(Porting from https://github.com/lightningnetwork/lnd/issues/8088 )

Problem

If remote party broadcasts a force close transaction, local funds are sent to an intermediate output, which then is swept to the wallet.

If the local node has lost everything but seed, it won't be able to recover funds through remote force close transactions. Otherwise it could find the other node and ask to force close the channel.

If the local node doesn't have channel.backup, but has channel.db, it may use chantools rescueclosed and recover funds. But still getting funds directly would be more convenient and more reliable.

Also the additional sweep transaction means more fees. Even if a node works well, it still has to pay this fee every time another node broadcasts a force close transaction.

Solution

I propose to use an address from the wallet directly instead of creating an intermediate output. If this is implemented, then only the party initiated the force closing will have to make a sweep transaction, while the passive party will receive its funds in the wallet directly. This will save fees and prevent loss of funds if only seed is available.

Anchors

I found this discussion https://github.com/lightningnetwork/lnd/issues/6855#issuecomment-1241655409

With STATIC_REMOTE_KEY channels (which is the default if you don't have anchor channels activated) the above is not necessary. But we never updated the sweeper logic (changes to the channel state machine are always high risk!) to skip the extra sweep. Also, the keys of the to_remote output aren't in the BIP84 key scope so they wouldn't be discovered by the wallet recovery if you recovered from seed. So it's still necessary to sweep them to safety (until we fix https://github.com/lightningnetwork/lnd/issues/4778).

With anchor output channels we now have a script in the to_remote, so it becomes crucial to sweep them again, as the script isn't included in the seed.

So, my proposal depends on fixing the wallet scanner to detect extra HD wallet derivation paths. Then such a path can be used for direct outputs of force close channels (one address per channel).

If a channel has anchor outputs, all non-anchor outputs must have 1 OP_CHECKSEQUENCEVERIFY (CSV). The wallet can have a path with such addresses! It is a normal P2WSH address, except it has 1 OP_CHECKSEQUENCEVERIFY (CSV) in the script. P2WSH witness scripts of such outputs currently look like this:

OP_PUSHBYTES_33 some_public_key
OP_CHECKSIGVERIFY
OP_PUSHNUM_1
OP_CSV

The public key is the only variable part. Private keys can be "stored" in seed using a separate HD wallet derivation path. If a channel is force closed by remote party, the local party doesn't have to do anything with these funds until it wants to spend them (e.g. open another channel).

Roasbeef commented 6 months ago

@starius I think if I were to do this from scratch today, I'd reach for Bitcoin Output Descriptors, as that's a general solution to this type of problem (importing a non-standard script into a wallet).

IMO the main argument for keeping things as is (always sweeping into a wallet addr), is that the special script won't be detected during normal rescans with just a seed. Though as you note, if you made a special case in the scanning code for this purpose (making addrs with that template during the look ahead computation), then we'd be able to pick them up like any other addr: https://github.com/btcsuite/btcwallet/blob/9a7dd2416f4ddad759330663d96ad98706d71f9e/wallet/recovery.go#L82-L117

lightningnetwork / lnd