search

getumbrel / umbrel-lightning

The official Lightning Node app for Umbrel, powered by LND.
https://umbrel.com
Other
15 stars 8 forks source link

Umbrel doesn't show On Chain balance after restore #20

Closed souspeed closed 8 months ago

souspeed commented 2 years ago

After standard restore procedure on Umbrel, only the lightning balances from the closed channels appear in Umbrel.

Umbrel does show the starting onchain balance 'deposit', however this whole balance is listed as 'lightning wallet - channel open', even when only part of this balance is used the open the channel. The remaining part is still in the onchain wallet. After restore and closing the channels, only the closed channel amount shows up in Umbrel 'deposit'. However; the remaining onchain balance remains in limbo.

I did perform the UTXO rescan procedure, but onchain balance didn't show.

Might be I did not wait long enough? 20+ hours after electrum finished indexing. Still I think this should be analysed/improved, as more and more 'beginners' will run into this issue.

Blue wallet did show the full balance within minutes. Maybe worth to check their process and mimic this logic on Umbrel for restore/re-scan purposes.

Guides https://community.getumbrel.com/t/how-to-restore-your-node-wallet/3602 https://bitcoin.cipix.eu/en/umbrel-troubleshooting/#umbrel-rescan-utxo

RaulRider commented 2 years ago

I have had the same problem when recovering from the seed and have tried several times.

Balance is either zero or inconsistent with the real UTXO (as I could check in Blue Wallet using the same 24 words seed).

NOTE: maybe not relevant, but I must say that every time I recovered from seed I copied an already downloaded blockchain but took the right steps to assure propoer permissions and oenwer of the files. The blockchain is synchronized 100% in all occasions. (scp, check permissions and owner, etc) and once copied you startup umbrel and shows 100% sinchronized.

I also waited for electrum server (electrs) to catch up and in the end the umbrel was fully sync’ed and running, but the balance is wrong

The only unexpected thing I can find is in electrs logs: This keeps showing in the logs although electrs rescan is completed

[2021-10-26T08:34:19.093Z WARN electrs::p2p] unexpected message: Unknown { command: CommandString("addr"), payload: [1, 48, 187, 119, 97, 9, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 165, 22, 227, 159, 71, 157] }

It is worth saying that:

Note: The only way to withdraw the total balance was with BW as umbrel gives a "not enough balance notice" due to worng balance.

synicasm commented 2 years ago

Just adding some notes to this issue, I experienced. I was having problems with my Umbrel PI install, so I wiped everything and started with a completely fresh install of Umbrel including a full redownload of the blockchain. I did not have any open lightning channels.

I used the recover seed to restore Umbrel when prompted and waited till the blockchain was 100% synced and 24 hours past 100% sync and Umbrel never showed my recovered wallet balance.
Using the seed phrase to recover the wallet in the mobile BlueWallet app my balance showed immediately.
Once I transferred the sats from my recovered BlueWallet to the recovered Umbrel wallet the stats mins the transaction fee appeared in my Umbrel wallet and were removed from my blue wallet. Keep in mind this is the same wallet with the same seed phrase. Now my Umbrel Wallet has a balance and my mobile Blue Walley has no balance.
Then I deleted my mobile BW and recovered again with the same seed phrase, now my balance in my mobile BW matches the balance in the Umbrel wallet.

Just adding there are a few people on the community site with this or similar issues.
https://community.getumbrel.com/t/recovery-from-seed-no-balance/2789 https://community.getumbrel.com/t/umbrel-bitcoin-wallet-empty-after-restore/4267/18

difigue commented 2 years ago

I have the same issue. Just wanted to add that because initially I was only testing Umbrel, I was syncing to a 1 TB HDD that I later found was failing. That means I went trough a lot of reflashes and builds from scratch since Umbrel v0.3.13. Everytime I entered my seed phrase to a brand new instance of Umbrel, my balance would appear within ~2 hours. Then I started having issues since v0.4.3 or v0.4.4.

I could get my balance to show again after waiting for a full blockchain and electrum sync and doing a rescan following this.

However, my electrs logs are flooding with the same warnings described by @RaulRider (unexpected message: Unknown). I don't know if this is a problem with electrs, lnd or umbrel, but I think it should be investigated since it's now a common issue many people are facing.

difigue commented 2 years ago

I don't know if this is a problem with electrs, lnd or umbrel, but I think it should be investigated since it's now a common issue many people are facing.

Now that I come across issue getumbrel/umbrel#1041 I wanted to add that maybe the problem is with neutrino as it handles your balance before the blockchain is even synced.

leonward commented 2 years ago

For what it's worth, I'm experiencing what looks like the same thing. A few notes in case it helps anyone:

fultonm commented 2 years ago

Same issue. As others have done, I restored with mnemonic from existing Umbrel install, then disconnected SSD and replaced the block and chainstate dirs with a fully synced node. The balance shows as 0, but there should be a small balance. Restoring with BlueWallet does show the balance. Now it appears electrs is catching up. I also have the unknown command "addr" warnings. Will update this message after electrs finishes indexing.

[2021-10-28T16:36:09.436Z INFO  electrs::index] indexing 2000 blocks: [318941..320940]
[2021-10-28T16:36:54.361Z WARN  electrs::p2p] unexpected message: Unknown { command: CommandString("addr"), payload: [1, 201, 207, 122, 97, 77, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 176, 9, 90, 182, 32, 141] }
[2021-10-28T16:37:31.709Z INFO  electrs::chain] chain updated: tip=000000000000000014095b9a2d326e985b37d949ff1575beb4aa63e3bac24fc8, height=320940
[2021-10-28T16:37:31.727Z INFO  electrs::index] indexing 2000 blocks: [320941..322940]

Re: neutrino-switcher, I get the following debug output:

docker logs -f neutrino-switcher
Checking if synced...
If set to neutrino then lets check bitcoind
Error: from bitcoind
{
  "code": -28,
  "message": "Loading block index…"
}

The bitcoind is showing as fully synced in Umbrel.

RaulRider commented 2 years ago

In my case I copied the blockchain too (copied blocks, chainstate and index directories using scp to preserve owner and presmission. It worked fine). But balance was still 0 even when synchronization was completed. Transactions done after spinning up the new restored umbrel node are correctly shown in the dashboard but this is not the case with the transaction history. My impression is that the issue may be related to the neutrino backend. It is a speculation, but it may be due to the fact that we used a copied blockchain and umbrel does not get the IBD (Initial Blockchain Download) as completed hence relaying on neutrino rather than bitcoindor electrs.

When setup a new umbrel node, lnd.conf has these lines

# Default to neutrino as the node is automatically switched to bitcoind once IBD is complete bitcoin.node=neutrino

Once IBD is completed, umbrel changes that line to bitcoin.node=bitcoind I will check if that is the case.

RaulRider commented 2 years ago

I wonder if running the script sudo ./scripts/configure will solve the issue. Configure script checks if blockchain is synced and then changes lnd.conf accordingly (neutrino to bitcoind)

# If node is already synced, do not reset to neutrino
if [[ -f "${STATUS_DIR}/node-status-bitcoind-ready" ]]; then
  sed -i "s/bitcoin.node=.*/bitcoin.node=bitcoind/g;" "$LND_CONF_FILE"
fi

Also, STATUS_DIR = "${UMBREL_ROOT}/statuses" so I will check if the file node-status-bitcoind-ready is there. If it is not, then umbrel is unaware of the blockchain fully synced and still uses neutrino.

leonward commented 2 years ago

Taking a quick look at the comments above.. Note that I decided to perform a reindex of the blockchain (bitcoin.conf: reindex=1)... because, well... it was a good idea at the time. So that's going to take me a long while.

Looking at the values of the content mentioned above:

I guess I'll just let it perform the reindex for a couple of days...

fultonm commented 2 years ago

I still have 0 balance after my electrs finally finished indexing blocks and is now caught up. Here's what I tried after the indexing was complete:

docker logs -f neutrino-switcher:

Checking if synced...
LND is already switched to bitcoind!
Checking not necessary. Exiting.

lnd/lnd.conf:

[Bitcoin]
bitcoin.active=1
bitcoin.mainnet=1
# Default to neutrino as the node is
# automatically switched to bitcoind once
# IBD is complete
bitcoin.node=bitcoind
bitcoin.defaultchanconfs=2

# [neutrino]
# Testnet neutrino peers that are automatically
# uncommented if Umbrel is configured for testnet
# neutrino.addpeer=testnet1-btcd.zaphq.io
# neutrino.addpeer=testnet2-btcd.zaphq.io

Sadly still showing 0 balance, while BlueWallet and my original Umbrel node (still running) is showing the correct value.

Debug log: https://pastebin.com/Y7ysW4TW

leonward commented 2 years ago

Still no joy. An update:

:shrug:

mbaniasad commented 2 years ago

Still no joy. An update:

  • I just performed a clean re-install using the exact recommend hardware (ssd, Pi psu etc) that is defined.
  • This time I did not copy the blockchain from another system to accelerate the blockchain sync. I waited about a week for it to complete a full sync of bitcoin & electrs, and then waited another 24 hours before checking.
  • The install was made on 0.4.7, and has been upgraded to 0.4.8.
  • I was confident my mnemonic was correct, since the existing funds appear in blue wallet, but to add confidence I sent it a few sats. Those new sats appeared in both the BlueWallet AND on the umbrel. Both wallets now show different balances, with umbrel only showing the new funds.

🤷

i have exactly reproduced this. except for sending the new sats part.

difigue commented 2 years ago

I hope this gets fixed soon.

synicasm commented 2 years ago

Have you tried this here https://bitcoin.cipix.eu/en/umbrel-troubleshooting/#umbrel-rescan-utxo

Doesn't fix the initial issue but might correct your balances.

guggero commented 2 years ago

I assume the on-chain balance is taken from lnd? Or what component has the on-chain wallet? If it's lnd, can anyone post their full lnd.log (from when the actual restore is performed)?

leonward commented 2 years ago

Here is the full lnd.log lnd.log

guggero commented 2 years ago

Thanks for the log. That was actually super helpful. Here's what I think is happening:

  1. The wallet was created (and with it recovery mode initiated) on 2021-11-11 18:55.
  2. At that moment, lnd is still in Neutrino mode and is trying to sync the chain backend.
  3. The sync is not happening for some reason (I assume it is related to what was fixed in https://github.com/lightningnetwork/lnd/issues/5962 which will be available in v0.14.0-beta that was just released). I also see errors related to Neutrino in the log (Banning peer ... reason=peer was unable to serve compact filters).
  4. The node was shut down on 2021-11-16 12:05, with it interrupting the wallet rescan (that process is currently NOT persisted, so if you shut down the node before it completes, it will not continue).
  5. The node was started again with bitcoind mode enabled on 2021-11-16 18:07.
  6. Because the rescan progress was aborted previously, only the last few blocks are re-scanned with the new bitcoind chain backend.

I think there are two ways to fix this:

  1. (temporary for users directly affected): Make sure the node is running on the bitcoind backend, then reset the wallet (by adding reset-wallet-transactions=true to the config).
  2. (more permanent): Upgrade to lnd v0.14.0-beta.
leonward commented 2 years ago

Thanks for the advice, I followed your first point since I didn't want to mess around with the container versions that are included on Umbrel. However after the reset-wallet-transactions=true flag was used, the balance shows up with the same incorrect value.

In the logs I can see that the rescan started at block height 706199 (as a side note, I don't know why that block was selected). All transactions to this wallet have taken place in blocks after that block. Here is an updated lnd.log in case it helps anyone.

lnd.log

guggero commented 2 years ago

Ah, I see. The reset-wallet-transactions flag doesn't actually enforce the recovery mode (where the wallet tries to detect used addresses) but instead only re-scans for outputs of already known addresses (of which there are none because the initial recovery was interrupted).

So I don't know if the following is possible in an Umbrel environment (or whether the middleware scripts will interfere), but this is what you can try:

That should find all addresses and restore your on-chain funds.

louneskmt commented 2 years ago

I'm afraid that this could break Umbrel login flow (wallet and account). Can you double check @lukechilds?

guggero commented 2 years ago

Or is there a way to restore lnd only without also nuking the bitcoind sync state (basically trying to avoid the Neutrino part completely if the chain backend is fully synced)?

difigue commented 2 years ago

I wonder, did you guys followed the steps on how to get your balance back pointed by an earlier comment? https://bitcoin.cipix.eu/en/umbrel-troubleshooting/#umbrel-rescan-utxo

Step 1 - Enter using SSH into your node:
Step 2 - execute the following command to rescan wallet’s UTXOs:
`sed -i "s/\[Application Options\]/\[Application Options\]\nreset-wallet-transactions=true/g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Step 3 - Wait some time so LND can rescan all wallet’s UTXOs. If it solve the issue, run this to stop it from rescanning on a future restart:
`sed -i "s/reset-wallet-transactions=true//g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Wait, it can take quite a while.

This worked for me and several others.

louneskmt commented 2 years ago

Usually, in order to reset user data, we ask users to run this:

sudo systemctl stop umbrel-startup && sudo rm -rf ~/umbrel/lnd/!(lnd.conf) && sudo rm ~/umbrel/db/user.json && sudo rm ~/umbrel/db/umbrel-seed/seed && sudo systemctl start umbrel-startup

It basically deletes everything except the config file in lnd directory and remove Umbrel-specific user files. Then they have to go through Umbrel setup process (new/restore seed, password).

I've asked two users to do that but the funds wouldn't recover either, after a seed restore.

leonward commented 2 years ago

I wonder, did you guys followed the steps on how to get your balance back pointed by an earlier comment? https://bitcoin.cipix.eu/en/umbrel-troubleshooting/#umbrel-rescan-utxo

Step 1 - Enter using SSH into your node:
Step 2 - execute the following command to rescan wallet’s UTXOs:
`sed -i "s/\[Application Options\]/\[Application Options\]\nreset-wallet-transactions=true/g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Step 3 - Wait some time so LND can rescan all wallet’s UTXOs. If it solve the issue, run this to stop it from rescanning on a future restart:
`sed -i "s/reset-wallet-transactions=true//g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Wait, it can take quite a while.

This worked for me and several others.

Yes, This was the approach I took, unfortunately it didn't fix it.

mbaniasad commented 2 years ago

I wonder, did you guys followed the steps on how to get your balance back pointed by an earlier comment? https://bitcoin.cipix.eu/en/umbrel-troubleshooting/#umbrel-rescan-utxo

Step 1 - Enter using SSH into your node:
Step 2 - execute the following command to rescan wallet’s UTXOs:
`sed -i "s/\[Application Options\]/\[Application Options\]\nreset-wallet-transactions=true/g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Step 3 - Wait some time so LND can rescan all wallet’s UTXOs. If it solve the issue, run this to stop it from rescanning on a future restart:
`sed -i "s/reset-wallet-transactions=true//g;" ~/umbrel/lnd/lnd.conf && sudo reboot`
Wait, it can take quite a while.

This worked for me and several others.

Yes, This was the approach I took, unfortunately it didn't fix it. Im having the same problem.

im doing the approach said here by @leonward anyways to see if I get different results. but it does not say how long it is going to take. so... waiting anybody knows how long it takes?

leonward commented 2 years ago

My 2nd lnd.log above should provide some timestamps for actual info, but from memory it was maybe 30-60mins. I

louneskmt commented 2 years ago

Hey guys, with the help of @guggero we might have pinpointed the issue. Apparently, when you recover your seed, LND enters in "Recovery Mode" and start address discovery, i.e. try to find every address of your wallet that has already been used and add it to the wallet known addresses. If somehow LND is stopped during this address discovery, the process won't resume on restart and LND won't know about some of your addresses. The fix is to recreate the wallet (recover) so LND can go through the address discovery process again. Correct me if I'm wrong @guggero.

1- First, remove the reset-wallet-transactions option that you may have set previously. 

sed -i '/reset-wallet-transactions=true/d;' ~/umbrel/lnd/lnd.conf

2- Reset your Umbrel user data 

sudo systemctl stop umbrel-startup && sudo rm -rf ~/umbrel/lnd/!(lnd.conf) && sudo rm -f ~/umbrel/db/user.json && sudo rm -f ~/umbrel/db/umbrel-seed/seed && sudo systemctl start umbrel-startup

3- Go through the setup process and recover from your seed.

4- Wait and do NOT restart or shutdown your Umbrel. This is very important as stopping LND during address discovery won't resume that process and LND won't know some of your addresses.

Warning: If after a little while, the Lightning wallet / widget seems to be endlessly loading and you see some RPC error, wallet not created in LND logs, restart from Step 2.

leonward commented 2 years ago

Yup, that solved it. Thinking back as to motivations of why I had stopped LND in case it helps think about changes in behavior / warnings / whatever.

Thanks!

souspeed commented 2 years ago

Thanks all & @guggero for taking care! Great to see progress happening.

julienPhA commented 2 years ago

@louneskmt Just because I'm interested: Is there any possibility to set the numer of adresses and the block height? Wallet recovery mode enabled with address lookahead of 250 addresses starting recovery of wallet from height=696889

Fogelman1982 commented 2 years ago

Hey guys, with the help of @guggero we might have pinpointed the issue. Apparently, when you recover your seed, LND enters in "Recovery Mode" and start address discovery, i.e. try to find every address of your wallet that has already been used and add it to the wallet known addresses. If somehow LND is stopped during this address discovery, the process won't resume on restart and LND won't know about some of your addresses. The fix is to recreate the wallet (recover) so LND can go through the address discovery process again. Correct me if I'm wrong @guggero.

1- First, remove the reset-wallet-transactions option that you may have set previously.

sed -i '/reset-wallet-transactions=true/d;' ~/umbrel/lnd/lnd.conf

2- Reset your Umbrel user data

sudo systemctl stop umbrel-startup && sudo rm -rf ~/umbrel/lnd/!(lnd.conf) && sudo rm -f ~/umbrel/db/user.json && sudo rm -f ~/umbrel/db/umbrel-seed/seed && sudo systemctl start umbrel-startup

3- Go through the setup process and recover from your seed.

4- Wait and do NOT restart or shutdown your Umbrel. This is very important as stopping LND during address discovery won't resume that process and LND won't know some of your addresses.

Warning: If after a little while, the Lightning wallet / widget seems to be endlessly loading and you see some RPC error, wallet not created in LND logs, restart from Step 2.

The second step don´t work for me, but as I am running Umbrel on Ubuntu does it mean I can remove all files EXCEPT lnd.conf in umbrel/lnd and also remove the two files user.json and seed and then sudu ./scripts/start to get the installation/recover process running?

louneskmt commented 2 years ago

@Fogelman1982 On a custom install, you can run the following command in your Umbrel root directory:

sudo scripts/stop && sudo rm -rf lnd/!(lnd.conf) && sudo rm -f db/user.json && sudo rm -f db/umbrel-seed/seed && sudo scripts/start
Fogelman1982 commented 2 years ago

@Fogelman1982 On a custom install, you can run the following command in your Umbrel root directory:

sudo scripts/stop && sudo rm -rf lnd/!(lnd.conf) && sudo rm -f db/user.json && sudo rm -f db/umbrel-seed/seed && sudo scripts/start

Thanks a lot! First funds shown up after a few minutes, need to wait some more before all funds are shown I guess ;)

magnusolof commented 2 years ago

@louneskmt I do not get to enter my seeds on step 3 "the wait thingy" is just spinning. What can i do now?

mrjuke commented 1 year ago

@Fogelman1982 On a custom install, you can run the following command in your Umbrel root directory:

sudo scripts/stop && sudo rm -rf lnd/!(lnd.conf) && sudo rm -f db/user.json && sudo rm -f db/umbrel-seed/seed && sudo scripts/start

I used this on my Linux install (0.5.1) and it didn’t worked. Did I do anything wrong? Please help me as I can’t restore my on chain wallet funds in umbrel

jimbrend commented 8 months ago

Hi!

Lots of people here!

Also, lots of funds being secured onchain... many new channels being opened! Update to latest version! Make sure to secure funds to avoid a disaster recovery or we can troubleshoot any current setups to get them up and running again, and since the last date on this issue we added an Automated Recovery feature for SCB backups!

We're streamlining these processes and helping make them easier, feel free to reach out to https://www.umbrel.com/support if anyone has not been able to yet recover for currently recommended steps,

We can assist 1:1 there!

Closing outdated thread