[upgrade 5.0 to 5.1] funds not back in wallet after channel closure

Sosthene00 commented 5 years ago

Background

After an update of BTCPay server that upgraded LND from 5.0 to 5.1, all my channels were inactive (see this issue). Since BTCPay's node is pruned, I copied the content of .lnd file inside another, unpruned, node and let LND do a full rescan. After the rescan I copied the data back in my BTCPay server. It appeared that most of my channels have been simultaneously force closed on the same block. One of these force closed channels had a positive balance on my side, but even as the closure appears to be settled funds are not in my wallet.

Here is the channel in closedchannels output:

"channel_point": "d1f7de5ff5770fb114b4fd1a458c5951af232b371b09379a7c44e1ffb9c6ebd0:1",
            "chan_id": "605221777524981761",
            "chain_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
            "closing_tx_hash": "ecfbd9cc8abd674ac57a0e9c7ef51a3a46bedd235a9f3c6b5805bcedd68aeada",
            "remote_pubkey": "02b414e4e29a685b8699152be47a0420fa3c8ab59629d2da2d6c0a95c582636350",
            "capacity": "1000000",
            "close_height": 555887,
            "settled_balance": "835541",
            "time_locked_balance": "0",
            "close_type": "REMOTE_FORCE_CLOSE"

The amount in settled balance does not show with walletbalance.

Just in case, here is the logs on startup:

2019-01-01 20:19:08.459 [INF] LTND: Version: 0.5.1-beta commit=basedon-v0.5.1-beta-dirty, build=production, logging=default
2019-01-01 20:19:08.460 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-01-01 20:19:08.473 [INF] CHDB: Checking for schema update: latest_version=7, db_version=7
2019-01-01 20:19:08.681 [INF] LTND: Primary chain is set to: bitcoin
2019-01-01 20:19:08.728 [INF] LTND: Initializing bitcoind backed fee estimator
2019-01-01 20:19:08.729 [INF] LNWL: Started listening for bitcoind block notifications via ZMQ on tcp://bitcoind:28332
2019-01-01 20:19:08.729 [INF] LNWL: Started listening for bitcoind transaction notifications via ZMQ on tcp://bitcoind:28333
2019-01-01 20:19:20.019 [INF] LNWL: Opened wallet
2019-01-01 20:19:21.866 [INF] LNWL: The wallet has been unlocked without a time limit
2019-01-01 20:19:21.957 [INF] LTND: LightningWallet opened
2019-01-01 20:19:22.051 [INF] HSWC: Restoring in-memory circuit state from disk
2019-01-01 20:19:22.056 [INF] HSWC: Payment circuits loaded: num_pending=0, num_open=0
2019-01-01 20:19:22.112 [INF] HSWC: Trimming open circuits for chan_id=545185:557:1, start_htlc_id=0
2019-01-01 20:19:22.113 [INF] HSWC: Trimming open circuits for chan_id=550263:409:0, start_htlc_id=0
2019-01-01 20:19:22.150 [INF] RPCS: RPC server listening on 127.0.0.1:10008
2019-01-01 20:19:22.151 [INF] RPCS: RPC server listening on [::]:10009
2019-01-01 20:19:22.154 [INF] RPCS: gRPC proxy started at [::]:8080
2019-01-01 20:19:22.160 [INF] LNWL: Started rescan from block 00000000000000000003edcd5a24914df0fe63e34b9a84cac416f037f8f9504f (height 556585) for 132 addresses
2019-01-01 20:19:22.165 [INF] LNWL: Starting rescan from block 00000000000000000003edcd5a24914df0fe63e34b9a84cac416f037f8f9504f
2019-01-01 20:19:22.180 [INF] LTND: Waiting for chain backend to finish sync, start_height=556585
2019-01-01 20:19:22.197 [INF] LNWL: Rescan finished at 556585 (00000000000000000003edcd5a24914df0fe63e34b9a84cac416f037f8f9504f)
2019-01-01 20:19:22.197 [INF] LNWL: Catching up block hashes to height 556585, this might take a while
2019-01-01 20:19:22.202 [INF] LNWL: Done catching up block hashes
2019-01-01 20:19:22.202 [INF] LNWL: Finished rescan for 132 addresses (synced to block 00000000000000000003edcd5a24914df0fe63e34b9a84cac416f037f8f9504f, height 556585)
2019-01-01 20:19:23.222 [INF] LTND: Chain backend is fully synced (end_height=556585)!
2019-01-01 20:19:23.274 [INF] HSWC: Starting HTLC Switch
2019-01-01 20:19:23.274 [INF] NTFN: New block epoch subscription
2019-01-01 20:19:23.274 [INF] NTFN: New block epoch subscription
2019-01-01 20:19:23.325 [INF] NTFN: New block epoch subscription
2019-01-01 20:19:23.376 [INF] CNCT: Creating ChannelArbitrators for 2 active channels
2019-01-01 20:19:23.377 [INF] NTFN: New block epoch subscription
2019-01-01 20:19:23.377 [INF] NTFN: New block epoch subscription

Your environment

version of lnd: 0.5.1-beta commit=basedon-v0.5.1-beta-dirty
which operating system (uname -a on *Nix): 4.15.0-43-generic #46-Ubuntu
version of btcd, bitcoind, or other backend: /Satoshi:0.17.0/

any other relevant environment details: LND is running inside a BTCPay server (docker). Here is the content of lnd.conf:

restlisten=0.0.0.0:8080
rpclisten=127.0.0.1:10008
rpclisten=0.0.0.0:10009
bitcoin.node=bitcoind
bitcoind.rpchost=bitcoind:43782
bitcoind.zmqpubrawblock=tcp://bitcoind:28332
bitcoind.zmqpubrawtx=tcp://bitcoind:28333
externalip=btcpay.sosthene.net:9735
alias=
adminmacaroonpath=/data/admin.macaroon
invoicemacaroonpath=/data/invoice.macaroon
readonlymacaroonpath=/data/readonly.macaroon
noseedbackup=1
notls=1
bitcoin.active=1
bitcoin.mainnet=1

Steps to reproduce

Upgrade LND with BTCPay, copy the data of .lnd to another node to allow LND to do a full rescan, copy the data back inside BTCPay.

Expected behaviour

Funds from a force closed channel should appear on my wallet.

Actual behaviour

The funds are nowhere to be seen.

halseth commented 5 years ago

Looking at the closing tx on a block explorer, It looks like it has been swept successfully, and the funds sent to the address bc1qp7cecfzq6k0s5syz62un6p09ur7t84957lk93t. These funds should be recognized during rescan, so check if you see them.

Sosthene00 commented 5 years ago

That's my point, I've already done a rescan and nothing showed up.

Hypothesis: maybe the tx hasn't been through yet when I've done my previous rescan, if I do a whole scan again it will eventually show up, does it make sense?

I want to be sure it might work before doing it again, because I've already done it twice, it took 24 hours each time and I don't like moving all those data around and messing with my other LND at home.

Since it's a very recent tx, isn't there a way to force a rescan from a said block_height instead of birthday block (which unfortunately is genesis block for my wallet)?

wpaulino commented 5 years ago

The wallet rescan when upgrading from 0.5 to 0.5.1 was done to fix an issue with the wallet not tracking UTXOs properly. This issue could have caused your wallet balance to be inaccurate. To ensure the funds have been returned to your wallet, you can check if the transaction (https://www.smartbit.com.au/tx/0211724bdb4b00c8fa71dbd56ac64b2cdd105b4b88085e755ec081e5ee66b003) exists within the wallet with lncli listchaintxns.

Sosthene00 commented 5 years ago

Hi, I can confirm that unfortunately the tx doesn't show up with lncli listchaintxns

I'll try to do a full rescan once again when I'll some time (except if there's a way to do only a partial rescan of course :smile: )

Roasbeef commented 5 years ago

Looking on chain, all outputs from that channel have been swept.

Roasbeef commented 5 years ago

If the server was very active, you may need to increase the look ahead distance for your rescan.

Sosthene00 commented 5 years ago

It wasn't very active afaik, and if by "increase the look ahead distance" you mean scanning earlier blocks, I already did a rescan from genesis block, not sure how to look further than that :sweat_smile:

halseth commented 5 years ago

By "very active" it means that you requested a lot of addresses that wasn't used.

For instance if you create 10 new addresses, and then only use the last of them, a look ahead distance of < 10 won't find the payment.

Roasbeef commented 5 years ago

Ping on issue @BobleChinois on chain, all the outputs have been swept.

Sosthene00 commented 5 years ago

Hi, sorry it has been quite busy these days. I don't remember creating that much addresses, but since it is on BTCPay I don't know all that happen under the hood I guess. I just checked it again, still nothing shows up. Is the "look ahead distance" something I can modify?

wpaulino commented 5 years ago

The look-ahead distance only applies when restoring from your seed and can be set with the lncli unlock --recovery_window, which I don't think is the case here?

Sosthene00 commented 5 years ago

Ok, point is, I don't think I will ever get the chance to use the unlock command, as BTCPay's LND wallet is unencrypted (and no seed too), and it seems by looking at the code this flag can't be used elsewhere, is it? Sorry for the headache guys!

wpaulino commented 5 years ago

Correct, and you won't be able to recover without a seed. Your best bet is just to run another manual rescan from the wallet birthday using the dropwtxmgr tool as we can see on-chain that the funds have been swept.

Sosthene00 commented 5 years ago

I don't get it, dropwtxmgr is a tool of btcwallet, right? I run LND with Core as a backend, should I be able to run this somehow?

halseth commented 5 years ago

LND uses btcwallet internally, so the tool can be used. Check maybe https://github.com/lightningnetwork/lnd/issues/1494#issuecomment-421908735

Sosthene00 commented 5 years ago

Ok I had a look at it, still not very clear how to run it and I have no time to tackle that right now anyway. Maybe we can close the issue for now and I'll reopen it later

Sosthene00 commented 5 years ago

Ok after doing the dropwtxmgr and running a full rescan again, the lost funds at last appeared in LND's wallet and are SAFU! Thanks everyone for your help, I think I'll write a detailed how-to of what I have done, just in case someone else run into the same trouble

lightningnetwork / lnd