search

lightningnetwork / lnd

Lightning Network Daemon ⚡️
MIT License
7.69k stars 2.08k forks source link

Not synced to chain but block height up to date #6301

Closed AZBTC21 closed 2 years ago

AZBTC21 commented 2 years ago

Background

Not synced to chain but block height up to date.

I also have a couple of forced closures in LIMBO status which are also not sweeping.

Your environment

Umbrel version

0.4.14

Flashed OS version

v0.4.14

Raspberry Pi Model

Revision : d03114 Serial : 10000000a274fa47 Model : Raspberry Pi 4 Model B Rev 1.4

Steps to reproduce

Issue: Not synced to chain but block height up to date

This issue started after doing the below transactions. Note: I did not make any changes to my node prior. My only relevant change other than channel fees was at least a week before when I upgraded the Umbrel OS to version 0.4.14:

cd ~/umbrel sudo ./scripts/stop sudo ./scripts/start

LND connection would not connect after above. LND shows ‘Running’, Lighting Wallet shows ‘Active’, but Channel Connections remain ‘0 Active’...

So tried to restart TOR and LND: cd ~/umbrel && docker-compose restart tor_server cd ~/umbrel && docker-compose restart lnd

Still no LND connection after above and same issue persists. Waited many hours to make sure I was being patient. Finally decided to restart: Restart from Umbrel Settings page. Had no success.

Next, manually backed up my lnd.conf and rebooted letting umbrel generate a new default one. I didn’t make any changes recently to my lnd.conf but wanted to make sure it wasn’t the issue and also to turn off my watch towers in case they were the issue as I saw some references in the logs. Still no luck.

Next, performed sudo systemctl stop umbrel-startup.service && docker system prune --force --all && sudo systemctl start umbrel-startup.service. No luck.

Next, reflashed my umbrel OS to the same version. No luck. Issue still persists.

I have reviewed all the logs related to Umbrel, LND, Bitcoin Core, and Electrs throughout the process. Here are some of the logs I posted in the Umbrel Community Telegram. https://t.me/getumbrel/161287. I also posted the issue on the Umbrel Community Troubleshooting Support page at https://community.getumbrel.com/t/lnd-channels-not-connecting-after-reboot/6535/6.

Based on my outputs of lncli getinfo and bitcoind getinfo I believe this issue might be related to an existing issue (https://github.com/lightningnetwork/lnd/issues/5132). Was a fix ever identified for this? I believe a few others have experienced this same problem.

Expected behaviour

LND channels connect. Would have thought it would say synch and remain in true status.

Actual behaviour

Bitcoin core seems to be running fine but LND channels will not connect.

Any assistance or guidance would be very much appreciated!

Thanks in advance.

lucasdcf commented 2 years ago

I understand that LND syncs to the current block height at startup. When you keep LND running, does it receive information about new blocks, or does it stay at the block height from the initial sync?

AZBTC21 commented 2 years ago

Hi lucasdcf - Thanks for your response. I just checked the LND log and it looks like the current blocks keep coming in. Here is the latest I see. Latest is block 725481 which is right at what mempool says. Thanks.

lnd | 2022-03-01 18:11:07.827 [INF] CNCT: ChannelArbitrator(f5f8e63f26f96ceb42eb6c29c18f3562f161e87f286f55f292b13fc9d7a04ba5:1): a contract has been fully resolved! lnd | 2022-03-01 18:11:07.827 [INF] CNCT: ChannelArbitrator(f5f8e63f26f96ceb42eb6c29c18f3562f161e87f286f55f292b13fc9d7a04ba5:1): still awaiting contract resolution lnd | 2022-03-01 18:21:27.800 [INF] NTFN: New block: height=725478, sha=000000000000000000092e9a3708812113b7e83624647884ae6b5b4b1c756ccb lnd | 2022-03-01 18:21:27.801 [INF] UTXN: Attempting to graduate height=725478: num_kids=0, num_babies=0 lnd | 2022-03-01 18:21:27.805 [INF] NTFN: Block disconnected from main chain: height=725478, sha=000000000000000000092e9a3708812113b7e83624647884ae6b5b4b1c756ccb lnd | 2022-03-01 18:21:28.183 [INF] NTFN: New block: height=725478, sha=000000000000000000092e9a3708812113b7e83624647884ae6b5b4b1c756ccb lnd | 2022-03-01 18:21:28.184 [INF] UTXN: Attempting to graduate height=725478: num_kids=0, num_babies=0 lnd | 2022-03-01 18:21:34.804 [INF] NTFN: New block: height=725479, sha=00000000000000000004ccf79d45f8ebfba4d92b6996fb387e6b25a63ad2db47 lnd | 2022-03-01 18:21:34.806 [INF] UTXN: Attempting to graduate height=725479: num_kids=0, num_babies=0 lnd | 2022-03-01 18:21:34.826 [INF] NTFN: Block disconnected from main chain: height=725479, sha=00000000000000000004ccf79d45f8ebfba4d92b6996fb387e6b25a63ad2db47 lnd | 2022-03-01 18:21:35.402 [INF] NTFN: New block: height=725479, sha=00000000000000000004ccf79d45f8ebfba4d92b6996fb387e6b25a63ad2db47 lnd | 2022-03-01 18:21:35.403 [INF] UTXN: Attempting to graduate height=725479: num_kids=0, num_babies=0 lnd | 2022-03-01 18:43:30.132 [INF] NTFN: New block: height=725480, sha=000000000000000000054a11541a0b304edf6f056cac7232128558503d936597 lnd | 2022-03-01 18:43:30.134 [INF] UTXN: Attempting to graduate height=725480: num_kids=0, num_babies=0 lnd | 2022-03-01 18:43:30.148 [INF] NTFN: Block disconnected from main chain: height=725480, sha=000000000000000000054a11541a0b304edf6f056cac7232128558503d936597 lnd | 2022-03-01 18:43:30.342 [INF] NTFN: New block: height=725480, sha=000000000000000000054a11541a0b304edf6f056cac7232128558503d936597 lnd | 2022-03-01 18:43:30.342 [INF] UTXN: Attempting to graduate height=725480: num_kids=0, num_babies=0 lnd | 2022-03-01 18:51:55.862 [INF] NTFN: New block: height=725481, sha=00000000000000000008a2ba69a88b09d8dd10b5c0e48121f1e660fbfd7e56aa lnd | 2022-03-01 18:51:55.863 [INF] UTXN: Attempting to graduate height=725481: num_kids=0, num_babies=0 lnd | 2022-03-01 18:51:55.867 [INF] NTFN: Block disconnected from main chain: height=725481, sha=00000000000000000008a2ba69a88b09d8dd10b5c0e48121f1e660fbfd7e56aa lnd | 2022-03-01 18:51:56.234 [INF] NTFN: New block: height=725481, sha=00000000000000000008a2ba69a88b09d8dd10b5c0e48121f1e660fbfd7e56aa lnd | 2022-03-01 18:51:56.235 [INF] UTXN: Attempting to graduate height=725481: num_kids=0, num_babies=0

lucasdcf commented 2 years ago

What does ~/umbrel/bin/bitcoin-cli -getinfo give you?

AZBTC21 commented 2 years ago

umbrel@umbrel:~/umbrel $ ~/umbrel/bin/bitcoin-cli -getinfo { "version": 220000, "blocks": 725483, "headers": 725483, "verificationprogress": 0.9999993584022417, "timeoffset": -4, "connections": { "in": 1, "out": 10, "total": 11 }, "proxy": "10.21.21.11:9050", "difficulty": 27967152532434.23, "chain": "main", "relayfee": 0.00001000, "warnings": "" }

lucasdcf commented 2 years ago

You can try an lnd rescan. If that doesn't work, we'll need to try bitcoind rescan.

lnd --reset-wallet-transactions

AZBTC21 commented 2 years ago

You can try an lnd rescan. If that doesn't work, we'll need to try bitcoind rescan.

lnd --reset-wallet-transactions

Hi Lucas. Where do I run this exactly? I tried from a few different directories and I am getting command not found. Thanks

lucasdcf commented 2 years ago

Sorry, I think Umbrel doesn't allow it. Change reset-wallet-transactions=true in your lnd.conf. (don't forget changing it back to false later, otherwise it will rescan every restart.

AZBTC21 commented 2 years ago

I updated the lnd.conf file with reset-wallet-transactions=true and restarted my lnd. How long should it take and what should I be on the lookout for? Thanks. Note: Will be sure to remember to change it back later.

AZBTC21 commented 2 years ago

Hi Lucas - Been watching the LND log. Looks like it just finished rescanning the wallet after about 7 hours. I'm not seeing any changes unfortunately... Zero LND channels are connecting even though I validated both bitcoin and LND block height is up-to-date and matches the mempool block. I'm also still seeing a few force closures with a couple in LIMBO which are not sweeping. Will remove the reset-wallet-transactions=true in my lnd.conf now so it does not rescan upon next restart in the future, as instructed. What next steps do you recommend? Thanks for your assistance.

lucasdcf commented 2 years ago

Are the errors in the lnd logs still the same?

I'm assuming you have txindex=1 (umbrel’s default) and that you're using A 1Tb disk with full Blockchain? Otherwise, the following will take a long time.

So I’d try to rescan Bitcoin now. For that you need to pass the rescan flag at startup. bitcoin --rescan

And you should adjust logging settings for lnd to get more detailed logs. We explain how to do it here: https://docs.lightning.engineering/lightning-network-tools/lnd/debugging_lnd Get the trace logs.

AZBTC21 commented 2 years ago

Apologies, as I am still learning. So thank you for taking it slow with me.

I am still seeing in /umbrel/bin/lncli getinfo "synced_to_chain": false, "synced_to_graph": false, Would have expected this to flip to 'true' and for my LND channels to connect.

I don't see any specific lnd errors that seem to be indicating the problem. All information in the logs are looking the same. But honestly I'm no expert in reading them. In addition, my Umbrel troubleshooting logs (see below) continue to say everything checks out okay. Yes, I am running Umbrel so I also assume the txindex=1 as I haven't changed anything like that. But let me know if there is a way to confirm. Yes, I am running a 1TB SSD drive. It has the full block chain with plenty of space. Sitting at 543GB now out of the 1TB. I think my channel.db is still manageable at 4.3GB but in the future when I can stabilize it I will work on compressing. Just haven't done so yet as I have been a minor routing node only running for about six months.

Can you please provide me a bit more detail on passing the bitcoin --rescan flag and where that goes? Does it also go in the lnd.conf? Will give that a try next with the logging turned on. Also, does this take a really long time to scan? I assume it is not a full new download of the chain, correct?

===================== = Umbrel debug info =

Umbrel version

0.4.15

Flashed OS version

v0.4.14

Raspberry Pi Model

Revision : d03114 Serial : 10000000a274fa47 Model : Raspberry Pi 4 Model B Rev 1.4

Firmware

Dec 1 2021 15:01:54 Copyright (c) 2012 Broadcom version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

Temperature

temp=50.6'C

Throttling

throttled=0x0

Memory usage

          total        used        free      shared  buff/cache   available

Mem: 7.8G 2.3G 169M 6.0M 5.4G 5.4G Swap: 4.1G 198M 3.9G

total: 29.0% bitcoin: 8.6% lnd: 8.2% electrs: 5.5% system: 4.4% bluewallet: 1.2% thunderhub: 0.4% ride-the-lightning: 0.4% tor: 0.3%

Memory monitor logs

1008 ? S 0:00 bash ./scripts/memory-monitor Memory monitor is already running 1008 ? S 0:00 bash ./scripts/memory-monitor Memory monitor is already running 1008 ? S 0:00 bash ./scripts/memory-monitor Memory monitor is already running 1008 ? S 0:00 bash ./scripts/memory-monitor Memory monitor is already running 1008 ? S 0:01 bash ./scripts/memory-monitor Memory monitor is already running

Filesystem information

Filesystem Size Used Avail Use% Mounted on /dev/root 59G 3.2G 53G 6% / /dev/sda1 916G 507G 363G 59% /home/umbrel/umbrel

Startup service logs

Mar 01 06:45:33 umbrel umbrel startup[988]: Creating neutrino-switcher ... done Mar 01 06:45:34 umbrel umbrel startup[988]: Removing status server iptables entry... Mar 01 06:45:34 umbrel umbrel startup[988]: Removed existing iptables entry. Mar 01 06:45:34 umbrel umbrel startup[988]: Starting installed apps... Mar 01 06:45:34 umbrel umbrel startup[988]: Starting app ride-the-lightning... Mar 01 06:45:34 umbrel umbrel startup[988]: Starting app bluewallet... Mar 01 06:45:34 umbrel umbrel startup[988]: Starting app thunderhub... Mar 01 06:45:40 umbrel umbrel startup[988]: Creating thunderhub_web_1 ... Mar 01 06:45:40 umbrel umbrel startup[988]: Creating ride-the-lightning_boltz_1 ... Mar 01 06:45:40 umbrel umbrel startup[988]: Creating ride-the-lightning_web_1 ... Mar 01 06:45:40 umbrel umbrel startup[988]: Creating ride-the-lightning_loop_1 ... Mar 01 06:45:41 umbrel umbrel startup[988]: Creating bluewallet_redis_1 ... Mar 01 06:45:45 umbrel umbrel startup[988]: Creating ride-the-lightning_boltz_1 ... done Mar 01 06:45:46 umbrel umbrel startup[988]: Creating thunderhub_web_1 ... done Mar 01 06:45:46 umbrel umbrel startup[988]: Creating bluewallet_redis_1 ... done Mar 01 06:45:46 umbrel umbrel startup[988]: Creating bluewallet_lndhub_1 ... Mar 01 06:45:46 umbrel umbrel startup[988]: Creating ride-the-lightning_web_1 ... done Mar 01 06:45:47 umbrel umbrel startup[988]: Creating ride-the-lightning_loop_1 ... done Mar 01 06:45:49 umbrel umbrel startup[988]: Creating bluewallet_lndhub_1 ... done Mar 01 06:45:50 umbrel umbrel startup[988]: Umbrel is now accessible at Mar 01 06:45:50 umbrel umbrel startup[988]: http://umbrel.local Mar 01 06:45:50 umbrel umbrel startup[988]: http://192.168.86.40 Mar 01 06:45:50 umbrel systemd[1]: Started Umbrel Startup Service. Mar 01 06:46:06 umbrel passwd[4849]: pam_unix(passwd:chauthtok): password changed for umbrel Mar 01 07:02:40 umbrel passwd[23367]: pam_unix(passwd:chauthtok): password changed for umbrel Mar 01 07:18:21 umbrel passwd[7738]: pam_unix(passwd:chauthtok): password changed for umbrel Mar 01 15:40:15 umbrel passwd[4384]: pam_unix(passwd:chauthtok): password changed for umbrel Mar 01 16:04:55 umbrel passwd[29364]: pam_unix(passwd:chauthtok): password changed for umbrel Mar 01 17:52:23 umbrel passwd[22173]: pam_unix(passwd:chauthtok): password changed for umbrel

External storage service logs

-- Logs begin at Tue 2022-03-01 06:44:47 UTC, end at Wed 2022-03-02 05:11:02 UTC. -- Mar 01 06:44:51 umbrel systemd[1]: Starting External Storage Mounter... Mar 01 06:44:52 umbrel external storage mounter[476]: Running external storage mount script... Mar 01 06:44:52 umbrel external storage mounter[476]: Found device "CT1000MX 500SSD1" Mar 01 06:44:52 umbrel external storage mounter[476]: Blacklisting USB device IDs against UAS driver... Mar 01 06:44:52 umbrel external storage mounter[476]: Rebinding USB drivers... Mar 01 06:44:52 umbrel external storage mounter[476]: Checking USB devices are back... Mar 01 06:44:52 umbrel external storage mounter[476]: Waiting for USB devices... Mar 01 06:44:53 umbrel external storage mounter[476]: Waiting for USB devices... Mar 01 06:44:54 umbrel external storage mounter[476]: Checking if the device is ext4... Mar 01 06:44:55 umbrel external storage mounter[476]: Yes, it is ext4 Mar 01 06:44:55 umbrel external storage mounter[476]: Checking if device contains an Umbrel install... Mar 01 06:44:55 umbrel external storage mounter[476]: Yes, it contains an Umbrel install Mar 01 06:44:55 umbrel external storage mounter[476]: Bind mounting external storage over local Umbrel installation... Mar 01 06:44:55 umbrel external storage mounter[476]: Bind mounting external storage over local Docker data dir... Mar 01 06:44:55 umbrel external storage mounter[476]: Bind mounting external storage to /swap Mar 01 06:44:55 umbrel external storage mounter[476]: Bind mounting SD card root at /sd-card... Mar 01 06:44:55 umbrel external storage mounter[476]: Checking Umbrel root is now on external storage... Mar 01 06:44:56 umbrel external storage mounter[476]: Checking /var/lib/docker is now on external storage... Mar 01 06:44:56 umbrel external storage mounter[476]: Checking /swap is now on external storage... Mar 01 06:44:56 umbrel external storage mounter[476]: Setting up swapfile Mar 01 06:44:56 umbrel external storage mounter[476]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes) Mar 01 06:44:56 umbrel external storage mounter[476]: no label, UUID=ea036c43-7526-4736-b85b-8411b9664e3d Mar 01 06:44:56 umbrel external storage mounter[476]: Checking SD Card root is bind mounted at /sd-root... Mar 01 06:44:56 umbrel external storage mounter[476]: Starting external drive mount monitor... Mar 01 06:44:56 umbrel external storage mounter[476]: Mount script completed successfully! Mar 01 06:44:56 umbrel systemd[1]: Started External Storage Mounter.

External storage SD card update service logs

-- Logs begin at Tue 2022-03-01 06:44:47 UTC, end at Wed 2022-03-02 05:11:02 UTC. -- Mar 01 06:45:07 umbrel systemd[1]: Starting External Storage SDcard Updater... Mar 01 06:45:07 umbrel external storage updater[910]: Checking if SD card Umbrel is newer than external storage... Mar 01 06:45:07 umbrel external storage updater[910]: No, SD version is not newer, exiting. Mar 01 06:45:07 umbrel systemd[1]: Started External Storage SDcard Updater.

Karen logs

./karen: line 68: /home/umbrel/umbrel/events/triggers/: Is a directory Got signal: karen is getting triggered! ./karen: line 68: /home/umbrel/umbrel/events/triggers/: Is a directory Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: debug karen is getting triggered! Debug result file generated Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: backup karen is getting triggered! Deriving keys... Creating backup... Adding random padding... 1+0 records in 1+0 records out 8427 bytes (8.4 kB, 8.2 KiB) copied, 0.000377071 s, 22.3 MB/s Creating encrypted tarball... backup/ backup/channel.backup backup/.padding Uploading backup... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 100 31856 0 0 100 31856 0 16319 0:00:01 0:00:01 --:--:-- 16319 100 32002 100 146 100 31856 60 13212 0:00:02 0:00:02 --:--:-- 13278 {"message":"Successfully uploaded backup 1646185039694.tar.gz.pgp for backup ID b88ebff8a3df2102455cc3164e2332d90cc8edfb27cca8d02824f87b17495fd1"}

====== Backup success =======

Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: change-password karen is getting triggered! New password: Retype new password: passwd: password updated successfully Got signal: debug karen is getting triggered! Debug result file generated Got signal: debug karen is getting triggered!

Docker containers

NAMES STATUS bluewallet_lndhub_1 Up 9 hours ride-the-lightning_loop_1 Up 9 hours ride-the-lightning_boltz_1 Up 9 hours thunderhub_web_1 Up 9 hours bluewallet_redis_1 Up 9 hours ride-the-lightning_web_1 Up 9 hours middleware Up 9 hours bitcoin Up 9 hours nginx Up 9 hours lnd Up 9 hours manager Up 9 hours umbrel_tor_server_1 Up 9 hours electrs Up 9 hours tor Up 9 hours dashboard Up 9 hours

Umbrel logs

Attaching to middleware, manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:40 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:41 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:42 GMT] "GET /v1/system/is-umbrel-os HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/system/get-update HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/system/memory HTTP/1.0" 200 202 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/system/storage HTTP/1.0" 200 202 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/system/temperature HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:44 GMT] "GET /v1/system/debug-result HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager manager | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:44 GMT] "GET /v1/external/price HTTP/1.0" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" manager | manager | umbrel-manager middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:23 GMT] "GET /v1/lnd/info/sync HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:24 GMT] "GET /v1/lnd/lightning/payments HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:24 GMT] "GET /v1/lnd/transaction HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:26 GMT] "GET /v1/lnd/channel HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:42 GMT] "GET /v1/bitcoind/info/sync HTTP/1.0" 200 86 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:42 GMT] "GET /v1/lnd/wallet/btc HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/lnd/lightning/invoices HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/lnd/info/sync HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/lnd/lightning/payments HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware middleware | ::ffff:10.21.21.2 - - [Wed, 02 Mar 2022 05:13:43 GMT] "GET /v1/lnd/transaction HTTP/1.0" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36" middleware | middleware | umbrel-middleware

Bitcoin Core logs

Attaching to bitcoin bitcoin | 2022-03-02T03:28:47Z UpdateTip: new best=00000000000000000002f3fc745a7bd360808bab78e557bbef8732d04e628e0a height=725528 version=0x20c00000 log2_work=93.377390 tx=714410517 date='2022-03-02T03:28:34Z' progress=1.000000 cache=87.1MiB(643360txo) bitcoin | 2022-03-02T03:34:09Z UpdateTip: new best=00000000000000000009ee34ee1e6c820cb43dcb834e592cb2cf41775fc2761a height=725529 version=0x2ec64004 log2_work=93.377404 tx=714412214 date='2022-03-02T03:33:48Z' progress=1.000000 cache=87.7MiB(648237txo) bitcoin | 2022-03-02T03:41:23Z Socks5() connect to 84.38.2.158:8333 failed: connection refused bitcoin | 2022-03-02T03:41:29Z UpdateTip: new best=00000000000000000000e3c339aedf0b642be33d84709c8f2b3958570bc84f22 height=725530 version=0x20600000 log2_work=93.377417 tx=714413051 date='2022-03-02T03:41:11Z' progress=1.000000 cache=88.1MiB(651114txo) bitcoin | 2022-03-02T03:46:12Z UpdateTip: new best=00000000000000000005654d0533a372aa8c903f3637e8bc05199395bb792d8b height=725531 version=0x20000004 log2_work=93.377431 tx=714413636 date='2022-03-02T03:46:22Z' progress=1.000000 cache=88.4MiB(653588txo) bitcoin | 2022-03-02T03:50:12Z UpdateTip: new best=00000000000000000002585d12c26168cbf24341f585f8db3f626168553f45fe height=725532 version=0x33de8000 log2_work=93.377444 tx=714414116 date='2022-03-02T03:49:43Z' progress=1.000000 cache=88.8MiB(656830txo) bitcoin | 2022-03-02T03:50:44Z New outbound peer connected: version: 70015, blocks=725532, peer=92 (block-relay-only) bitcoin | 2022-03-02T03:52:25Z Socks5() connect to 2800:484:587b:d200::5:8333 failed: general failure bitcoin | 2022-03-02T03:58:00Z UpdateTip: new best=0000000000000000000725134a6c985a3fafd28b853bc1915f4dcd72e9f323a1 height=725533 version=0x2b91a000 log2_work=93.377458 tx=714414948 date='2022-03-02T03:57:39Z' progress=1.000000 cache=89.1MiB(659258txo) bitcoin | 2022-03-02T04:02:17Z Socks5() connect to 195.133.29.230:8333 failed: general failure bitcoin | 2022-03-02T04:08:35Z New outbound peer connected: version: 70016, blocks=725533, peer=94 (block-relay-only) bitcoin | 2022-03-02T04:14:19Z UpdateTip: new best=00000000000000000007d48dce90d2a7d15972a11e0491feb9db6f9bb254a770 height=725534 version=0x20000000 log2_work=93.377471 tx=714416389 date='2022-03-02T04:14:06Z' progress=1.000000 cache=90.0MiB(666996txo) bitcoin | 2022-03-02T04:16:22Z Socks5() connect to 84.240.99.143:8333 failed: general failure bitcoin | 2022-03-02T04:17:31Z UpdateTip: new best=000000000000000000028319cfe61275dd9b12be5d6613f778f3c575f134eda7 height=725535 version=0x20000004 log2_work=93.377484 tx=714417902 date='2022-03-02T04:17:28Z' progress=1.000000 cache=90.6MiB(671825txo) bitcoin | 2022-03-02T04:19:26Z Socks5() connect to 84.17.60.16:8333 failed: general failure bitcoin | 2022-03-02T04:34:48Z UpdateTip: new best=00000000000000000008bc014451b4991c3e5cf0d5f5539820409b8e637aa897 height=725536 version=0x20004000 log2_work=93.377498 tx=714420019 date='2022-03-02T04:34:38Z' progress=1.000000 cache=91.8MiB(681000txo) bitcoin | 2022-03-02T04:37:59Z Socks5() connect to 88.74.223.131:8333 failed: general failure bitcoin | 2022-03-02T04:40:52Z New outbound peer connected: version: 70016, blocks=725536, peer=97 (block-relay-only) bitcoin | 2022-03-02T04:41:58Z Socks5() connect to 92.195.91.142:8333 failed: host unreachable bitcoin | 2022-03-02T04:45:28Z UpdateTip: new best=000000000000000000064b20091db3c52f10eca226251cfcf2cbdade6206a8f9 height=725537 version=0x20800004 log2_work=93.377511 tx=714423937 date='2022-03-02T04:44:52Z' progress=1.000000 cache=92.7MiB(688616txo) bitcoin | 2022-03-02T04:45:47Z Socks5() connect to 2600:3c01::f03c:91ff:fed8:db38:8333 failed: general failure bitcoin | 2022-03-02T04:48:50Z UpdateTip: new best=000000000000000000051bef222177d5e1c8439780ad2de357b2f93ce05d0b0b height=725538 version=0x20c00000 log2_work=93.377525 tx=714425670 date='2022-03-02T04:48:29Z' progress=1.000000 cache=92.8MiB(689328txo) bitcoin | 2022-03-02T04:50:09Z New outbound peer connected: version: 70015, blocks=725538, peer=98 (block-relay-only) bitcoin | 2022-03-02T04:54:53Z UpdateTip: new best=00000000000000000004147efa0a7a5c55620d015b6b30e1c4526a7a3a98bda8 height=725539 version=0x27ffe004 log2_work=93.377538 tx=714427283 date='2022-03-02T04:54:34Z' progress=1.000000 cache=93.1MiB(691673txo) bitcoin | 2022-03-02T04:56:01Z Socks5() connect to 128.71.87.3:8333 failed: general failure bitcoin | 2022-03-02T04:59:55Z New outbound peer connected: version: 70016, blocks=725539, peer=99 (block-relay-only) bitcoin | 2022-03-02T05:09:48Z UpdateTip: new best=000000000000000000055a95d9661726e503824c516e2704884e398d605dfbf0 height=725540 version=0x2fffe004 log2_work=93.377552 tx=714429712 date='2022-03-02T05:09:13Z' progress=1.000000 cache=94.6MiB(703723txo) bitcoin | 2022-03-02T05:10:31Z Socks5() connect to 2a01:4f8:c0c:ead9::1:8333 failed: general failure bitcoin | 2022-03-02T05:13:35Z New outbound peer connected: version: 70015, blocks=725540, peer=102 (block-relay-only)

LND logs

Attaching to lnd lnd | 2022-03-02 04:17:33.578 [INF] NTFN: New block: height=725535, sha=000000000000000000028319cfe61275dd9b12be5d6613f778f3c575f134eda7 lnd | 2022-03-02 04:17:33.581 [INF] UTXN: Attempting to graduate height=725535: num_kids=0, num_babies=0 lnd | 2022-03-02 04:17:33.602 [INF] NTFN: Block disconnected from main chain: height=725535, sha=000000000000000000028319cfe61275dd9b12be5d6613f778f3c575f134eda7 lnd | 2022-03-02 04:17:33.881 [INF] NTFN: New block: height=725535, sha=000000000000000000028319cfe61275dd9b12be5d6613f778f3c575f134eda7 lnd | 2022-03-02 04:17:33.881 [INF] UTXN: Attempting to graduate height=725535: num_kids=0, num_babies=0 lnd | 2022-03-02 04:34:50.376 [INF] NTFN: New block: height=725536, sha=00000000000000000008bc014451b4991c3e5cf0d5f5539820409b8e637aa897 lnd | 2022-03-02 04:34:50.377 [INF] UTXN: Attempting to graduate height=725536: num_kids=0, num_babies=0 lnd | 2022-03-02 04:34:50.382 [INF] NTFN: Block disconnected from main chain: height=725536, sha=00000000000000000008bc014451b4991c3e5cf0d5f5539820409b8e637aa897 lnd | 2022-03-02 04:34:50.802 [INF] NTFN: New block: height=725536, sha=00000000000000000008bc014451b4991c3e5cf0d5f5539820409b8e637aa897 lnd | 2022-03-02 04:34:50.803 [INF] UTXN: Attempting to graduate height=725536: num_kids=0, num_babies=0 lnd | 2022-03-02 04:45:32.095 [INF] NTFN: New block: height=725537, sha=000000000000000000064b20091db3c52f10eca226251cfcf2cbdade6206a8f9 lnd | 2022-03-02 04:45:32.096 [INF] UTXN: Attempting to graduate height=725537: num_kids=0, num_babies=0 lnd | 2022-03-02 04:45:32.105 [INF] NTFN: Block disconnected from main chain: height=725537, sha=000000000000000000064b20091db3c52f10eca226251cfcf2cbdade6206a8f9 lnd | 2022-03-02 04:45:32.703 [INF] NTFN: New block: height=725537, sha=000000000000000000064b20091db3c52f10eca226251cfcf2cbdade6206a8f9 lnd | 2022-03-02 04:45:32.704 [INF] UTXN: Attempting to graduate height=725537: num_kids=0, num_babies=0 lnd | 2022-03-02 04:48:52.623 [INF] NTFN: New block: height=725538, sha=000000000000000000051bef222177d5e1c8439780ad2de357b2f93ce05d0b0b lnd | 2022-03-02 04:48:52.624 [INF] UTXN: Attempting to graduate height=725538: num_kids=0, num_babies=0 lnd | 2022-03-02 04:48:52.629 [INF] NTFN: Block disconnected from main chain: height=725538, sha=000000000000000000051bef222177d5e1c8439780ad2de357b2f93ce05d0b0b lnd | 2022-03-02 04:48:53.095 [INF] NTFN: New block: height=725538, sha=000000000000000000051bef222177d5e1c8439780ad2de357b2f93ce05d0b0b lnd | 2022-03-02 04:48:53.096 [INF] UTXN: Attempting to graduate height=725538: num_kids=0, num_babies=0 lnd | 2022-03-02 04:54:55.371 [INF] NTFN: New block: height=725539, sha=00000000000000000004147efa0a7a5c55620d015b6b30e1c4526a7a3a98bda8 lnd | 2022-03-02 04:54:55.372 [INF] UTXN: Attempting to graduate height=725539: num_kids=0, num_babies=0 lnd | 2022-03-02 04:54:55.383 [INF] NTFN: Block disconnected from main chain: height=725539, sha=00000000000000000004147efa0a7a5c55620d015b6b30e1c4526a7a3a98bda8 lnd | 2022-03-02 04:54:55.628 [INF] NTFN: New block: height=725539, sha=00000000000000000004147efa0a7a5c55620d015b6b30e1c4526a7a3a98bda8 lnd | 2022-03-02 04:54:55.628 [INF] UTXN: Attempting to graduate height=725539: num_kids=0, num_babies=0 lnd | 2022-03-02 05:09:49.920 [INF] NTFN: New block: height=725540, sha=000000000000000000055a95d9661726e503824c516e2704884e398d605dfbf0 lnd | 2022-03-02 05:09:49.920 [INF] UTXN: Attempting to graduate height=725540: num_kids=0, num_babies=0 lnd | 2022-03-02 05:09:49.925 [INF] NTFN: Block disconnected from main chain: height=725540, sha=000000000000000000055a95d9661726e503824c516e2704884e398d605dfbf0 lnd | 2022-03-02 05:09:50.339 [INF] NTFN: New block: height=725540, sha=000000000000000000055a95d9661726e503824c516e2704884e398d605dfbf0 lnd | 2022-03-02 05:09:50.339 [INF] UTXN: Attempting to graduate height=725540: num_kids=0, num_babies=0

electrs logs

Attaching to electrs electrs | [2022-03-02T04:13:24.146Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: a5b5bf99ca529b03af8d630d6df574e26b33d142cb77fba69b41ed221acd918e electrs | [2022-03-02T04:13:24.158Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 16b55dc9593907c21ce4141ca98bfd4c8d5608aad06366d43908119ea37923b6 electrs | [2022-03-02T04:13:24.169Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 51d6877dc8dd25b58126443efd92fb33e3079798939bf356a15670d2722379b9 electrs | [2022-03-02T04:13:24.180Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 0f121a0ee7426a5adf737c98efb7e1092b04cd233206310be23048d15803d52a electrs | [2022-03-02T04:13:24.192Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 273e92bb34492864518808f5989de168b355f20f56cdbb72277bf9517e703d3b electrs | [2022-03-02T04:13:24.224Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: cddd046b8159f26d9e6819893aabfa0941acbf02b94657e6faab7cee2815539b electrs | [2022-03-02T04:13:24.257Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: e68775dfcbddc1a8278018eab049861a9864e8981b8616e33d0444287c99560c electrs | [2022-03-02T04:13:24.268Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 1dbb77a14c55c0c92fcd85d99b236e84fb8df50870eafa17595e957b73c40663 electrs | [2022-03-02T04:13:24.300Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 874b3da750a5e61525811ed76a1bdc1712fd34a9598c7264ef2d926305a0afe7 electrs | [2022-03-02T04:13:24.333Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 254eccf3573a3e442bbbffbe5b4ea22efe420712b4fcad08ed62752e8fca6131 electrs | [2022-03-02T04:13:24.344Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 5e9dead76c0847b49ad25b20f83f178484f9e87e0b8e24de0ad8e58e53b41c83 electrs | [2022-03-02T04:13:24.377Z ERROR electrs::server] 0: disconnecting due to failed to send response: "{\"id\":566,\"jsonrpc\":\"2.0\",\"result\":{\"confirmed\":3474472,\"unconfirmed\":0}}\n" electrs | [2022-03-02T04:13:24.377Z WARN electrs::server] 0: failed to shutdown TCP connection Transport endpoint is not connected (os error 107) electrs | [2022-03-02T04:14:19.798Z INFO electrs::index] indexing 1 blocks: [725534..725534] electrs | [2022-03-02T04:14:20.001Z INFO electrs::chain] chain updated: tip=00000000000000000007d48dce90d2a7d15972a11e0491feb9db6f9bb254a770, height=725534 electrs | [2022-03-02T04:14:29.329Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 51e1f6c3e06f1ec0f4f79ab452a23046a5fc3a7955ff47b58de926b54191e201 electrs | [2022-03-02T04:14:34.270Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 51e1f6c3e06f1ec0f4f79ab452a23046a5fc3a7955ff47b58de926b54191e201 electrs | [2022-03-02T04:14:39.214Z INFO electrs::electrum] your wallet uses less efficient method of querying electrs, consider contacting the developer of your wallet. Reason: blockchain.scripthash.get_balance called for unsubscribed scripthash: 51e1f6c3e06f1ec0f4f79ab452a23046a5fc3a7955ff47b58de926b54191e201 electrs | [2022-03-02T04:17:32.131Z INFO electrs::index] indexing 1 blocks: [725535..725535] electrs | [2022-03-02T04:17:32.345Z INFO electrs::chain] chain updated: tip=000000000000000000028319cfe61275dd9b12be5d6613f778f3c575f134eda7, height=725535 electrs | [2022-03-02T04:34:49.028Z INFO electrs::index] indexing 1 blocks: [725536..725536] electrs | [2022-03-02T04:34:49.151Z INFO electrs::chain] chain updated: tip=00000000000000000008bc014451b4991c3e5cf0d5f5539820409b8e637aa897, height=725536 electrs | [2022-03-02T04:45:29.232Z INFO electrs::index] indexing 1 blocks: [725537..725537] electrs | [2022-03-02T04:45:29.389Z INFO electrs::chain] chain updated: tip=000000000000000000064b20091db3c52f10eca226251cfcf2cbdade6206a8f9, height=725537 electrs | [2022-03-02T04:48:50.604Z INFO electrs::index] indexing 1 blocks: [725538..725538] electrs | [2022-03-02T04:48:50.726Z INFO electrs::chain] chain updated: tip=000000000000000000051bef222177d5e1c8439780ad2de357b2f93ce05d0b0b, height=725538 electrs | [2022-03-02T04:54:53.868Z INFO electrs::index] indexing 1 blocks: [725539..725539] electrs | [2022-03-02T04:54:53.990Z INFO electrs::chain] chain updated: tip=00000000000000000004147efa0a7a5c55620d015b6b30e1c4526a7a3a98bda8, height=725539 electrs | [2022-03-02T05:09:48.550Z INFO electrs::index] indexing 1 blocks: [725540..725540] electrs | [2022-03-02T05:09:48.672Z INFO electrs::chain] chain updated: tip=000000000000000000055a95d9661726e503824c516e2704884e398d605dfbf0, height=725540

Tor logs

Attaching to umbrel_tor_server_1, tor tor_server_1 | Mar 02 02:24:24.000 [notice] Heartbeat: Tor's uptime is 6:00 hours, with 79 circuits open. I've sent 18.04 MB and received 12.26 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 19 connections with IPv4 and 0 with IPv6. tor_server_1 | Mar 02 02:24:24.000 [notice] While bootstrapping, fetched this many bytes: 663800 (consensus network-status fetch); 27237 (authority cert fetch); 5390793 (microdescriptor fetch) tor_server_1 | Mar 02 02:24:24.000 [notice] While not bootstrapping, fetched this many bytes: 160001 (consensus network-status fetch); 247513 (microdescriptor fetch) tor_server_1 | Mar 02 02:24:24.000 [notice] Heartbeat: Tor's uptime is 6:00 hours, with 71 circuits open. I've sent 17.09 MB and received 11.50 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 12 connections with IPv4 and 0 with IPv6. tor_server_1 | Mar 02 02:24:24.000 [notice] While bootstrapping, fetched this many bytes: 663814 (consensus network-status fetch); 27237 (authority cert fetch); 5404222 (microdescriptor fetch) tor_server_1 | Mar 02 02:24:24.000 [notice] While not bootstrapping, fetched this many bytes: 152594 (consensus network-status fetch); 230734 (microdescriptor fetch) tor_server_1 | Mar 02 02:55:04.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 160 buildtimes. tor | Mar 02 03:28:30.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:02:17.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:16:22.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:19:26.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:37:59.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:41:58.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up. tor | Mar 02 04:56:01.000 [notice] Have tried resolving or connecting to address '[scrubbed]' at 3 different places. Giving up.

App logs

bluewallet

lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() redis_1 | 8:M 02 Mar 2022 05:12:58.112 * Background saving terminated with success lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated lndhub_1 | updateLightning() lndhub_1 | updated

ride-the-lightning

loop_1 | 2022-03-01 20:27:48.988 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009 web_1 | [3/1/2022, 8:24:53 PM] INFO: CSRF => Setting up CSRF... loop_1 | 2022-03-01 20:27:48.996 [INF] LNDC: Connected to lnd web_1 | loop_1 | 2022-03-01 20:27:48.998 [INF] LNDC: Waiting for lnd to unlock web_1 | [3/1/2022, 8:24:53 PM] INFO: CSRF => CSRF Set. loop_1 | loopd exited with an error: rpc error: code = Unknown desc = waiting to start, RPC services not available web_1 | loop_1 | 2022-03-01 20:28:50.699 [INF] LOOPD: Version: 0.12.1-beta commit= web_1 | [3/1/2022, 8:24:53 PM] INFO: App => Setting up Application Routes... loop_1 | 2022-03-01 20:28:50.700 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009 web_1 | loop_1 | 2022-03-01 20:28:50.704 [INF] LNDC: Connected to lnd web_1 | [3/1/2022, 8:24:53 PM] INFO: App => Application Routes Set. loop_1 | 2022-03-01 20:28:50.704 [INF] LNDC: Waiting for lnd to unlock web_1 | loop_1 | loopd exited with an error: rpc error: code = Unknown desc = wallet locked, unlock it to enable full RPC access loop_1 | 2022-03-01 20:29:52.396 [INF] LOOPD: Version: 0.12.1-beta commit= web_1 | [3/1/2022, 8:24:53 PM] INFO: WebSocketServer => Connecting Websocket Server... loop_1 | 2022-03-01 20:29:52.397 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009 web_1 | loop_1 | 2022-03-01 20:29:52.398 [INF] LNDC: Connected to lnd web_1 | [3/1/2022, 8:24:53 PM] INFO: WebSocketServer => Websocket Server Connected. loop_1 | 2022-03-01 20:29:52.398 [INF] LNDC: Waiting for lnd to unlock web_1 | loop_1 | 2022-03-01 20:29:53.242 [INF] LNDC: lnd version: v0.14.2-beta, build tags 'autopilotrpc,signrpc,walletrpc,chainrpc,invoicesrpc,watchtowerrpc,monitoring,kvdb_postgres,kvdb_etcd' web_1 | [3/1/2022, 8:24:53 PM] INFO: RTL => Server is up and running, please open the UI at http://localhost:3001. loop_1 | 2022-03-01 20:29:53.243 [INF] LNDC: Using network mainnet web_1 | loop_1 | 2022-03-01 20:29:53.243 [INF] LNDC: Waiting for lnd to be fully synced to its chain backend, this might take a while

thunderhub

Attaching to thunderhub_web_1 web_1 | web_1 | > thunderhub@0.12.31 start /app web_1 | > next start web_1 | web_1 | ready - started server on 0.0.0.0:3000, url: http://localhost:3000

==== Result ====

The debug script did not automatically detect any issues with your Umbrel.

lucasdcf commented 2 years ago

It won't be a full download of the chain if you have txindex=1. If you're using Umbrel's default, it's ok. It can take a few hours tho.

For rescanning, add rescan=1 to bitcoin.conf (again, remove it later)

AZBTC21 commented 2 years ago

I have added rescan=1 to bitcoin.conf and rebooted the umbrel. To be honest, I am not sure if this one is working as opposed to the last rescan (lnd). I'm reviewing the logs and keep seeing the latest blocks posting in the lnd and bitcoin logs. Can you kindly guide me on where I should be looking to validate it is actually doing the bitcoin rescan? Understand it will take a few hours, which is fine.

I updated the logs to trace as well. Hoping that will help shed some light.

If this doesn't work I feel it is probably time to just try and close out all my channels and restore my funds. I have my seed and SCB. If you have instructions on that for an Umbrel which you could guide me to it would be appreciated.

Thanks for all your help!

AZBTC21 commented 2 years ago

By the way, if I need to recover my major concern is the current status of a few force closures in LIMBO status with no block height (it is empty). How can I be sure to get these funds to sweep and/or are these at risk? Thanks.

lucasdcf commented 2 years ago

Try one of the following in your terminal when starting bitcoind. (I'm not an Umbrel user, so I'm not sure) bitcoind --rescan bitcoin-qt --rescan bitcoin-core --rescan bitcoin-cli rescanblockchain

Don't forget to get the trace logs for lnd. (debug logs for bitcoind would be good too)

lucasdcf commented 2 years ago

If the rescan doesn't work, you don't need to recover using SCB, you can just force close all your channels (if you have an updated channel.db that will be ok) The force closes are in the blockchain, right? You'll get the funds once the timelocks expire. You don't see the block height probably for the same reason your channels are not connecting.

But let's see the logs for the rescan first. tail -f lnd.log tail -f ~/umbrel/bitcoin/debug.log

AZBTC21 commented 2 years ago

Here is what I am seeing at the moment. Probably not too helpful. I believe I turned on the trace and do see some with TRC in the last log below...

tail -f ~/umbrel/lnd/logs/bitcoin/mainnet/lnd.log:

umbrel@umbrel:~ $ tail -f ~/umbrel/lnd/logs/bitcoin/mainnet/lnd.log 2022-03-03 01:32:26.745 [INF] NTFN: New block: height=725661, sha=00000000000000000002d6a54e4f4f2c03b52949e98566398373e4b878a51813 2022-03-03 01:32:26.746 [INF] UTXN: Attempting to graduate height=725661: num_kids=0, num_babies=0 2022-03-03 01:32:26.751 [INF] NTFN: Block disconnected from main chain: height=725661, sha=00000000000000000002d6a54e4f4f2c03b52949e98566398373e4b878a51813 2022-03-03 01:32:27.217 [INF] NTFN: New block: height=725661, sha=00000000000000000002d6a54e4f4f2c03b52949e98566398373e4b878a51813 2022-03-03 01:32:27.217 [INF] UTXN: Attempting to graduate height=725661: num_kids=0, num_babies=0 2022-03-03 01:43:32.848 [INF] NTFN: New block: height=725662, sha=00000000000000000005f19cf70f2d0cce8441188afe0c135a286e6db0e3a2f9 2022-03-03 01:43:32.849 [INF] UTXN: Attempting to graduate height=725662: num_kids=0, num_babies=0 2022-03-03 01:43:32.853 [INF] NTFN: Block disconnected from main chain: height=725662, sha=00000000000000000005f19cf70f2d0cce8441188afe0c135a286e6db0e3a2f9 2022-03-03 01:43:33.055 [INF] NTFN: New block: height=725662, sha=00000000000000000005f19cf70f2d0cce8441188afe0c135a286e6db0e3a2f9 2022-03-03 01:43:33.056 [INF] UTXN: Attempting to graduate height=725662: num_kids=0, num_babies=0

tail -f ~/umbrel/bitcoin/debug.log

umbrel@umbrel:~ $ tail -f ~/umbrel/bitcoin/debug.log 2022-03-03T01:28:55Z New outbound peer connected: version: 70016, blocks=725660, peer=36 (outbound-full-relay) 2022-03-03T01:29:04Z Socks5() connect to iaxjd6cks2hzuv3b2fudvs6r7m7tpik2xojac765ysg4crckvtsvkkid.onion:8333 failed: host unreachable 2022-03-03T01:29:05Z New outbound peer connected: version: 70015, blocks=725660, peer=37 (outbound-full-relay) 2022-03-03T01:30:32Z New outbound peer connected: version: 70016, blocks=725660, peer=39 (outbound-full-relay) 2022-03-03T01:32:24Z UpdateTip: new best=00000000000000000002d6a54e4f4f2c03b52949e98566398373e4b878a51813 height=725661 version=0x20c00000 log2_work=93.379181 tx=714696226 date='2022-03-03T01:32:16Z' progress=1.000000 cache=22.1MiB(163262txo) 2022-03-03T01:32:59Z New outbound peer connected: version: 70016, blocks=725661, peer=40 (block-relay-only) 2022-03-03T01:34:43Z New outbound peer connected: version: 70016, blocks=725661, peer=41 (block-relay-only) 2022-03-03T01:37:36Z Socks5() connect to 104.168.244.16:8333 failed: general failure 2022-03-03T01:43:31Z UpdateTip: new best=00000000000000000005f19cf70f2d0cce8441188afe0c135a286e6db0e3a2f9 height=725662 version=0x3335e000 log2_work=93.379194 tx=714697771 date='2022-03-03T01:42:44Z' progress=1.000000 cache=22.8MiB(169074txo) 2022-03-03T01:45:38Z Socks5() connect to 137.164.229.162:8333 failed: connection refused

Seeing some TRC's below which I believe might be the trace:

2022-03-02 23:22:37.221 [DBG] UTXN: NurseryReport: building nursery report for channel 5742578a98a7fb7c79ef8fcba85276c178f66320e636f02266ab9f5631f44310:0 2022-03-02 23:22:37.224 [DBG] UTXN: NurseryReport: building nursery report for channel 93ab6c3c8de888d8016dc9ec0dea20e486689f1892667293016395d53ad8e976:0 2022-03-02 23:22:37.227 [DBG] UTXN: NurseryReport: building nursery report for channel f5f8e63f26f96ceb42eb6c29c18f3562f161e87f286f55f292b13fc9d7a04ba5:1 2022-03-02 23:22:37.229 [DBG] UTXN: NurseryReport: building nursery report for channel 694d0f356d0ea610461f23f7164fc964dd97a822a357349db0d5f4eeff3087f3:1 2022-03-02 23:22:37.627 [DBG] RPCS: [ListPayments] 2022-03-02 23:22:49.576 [INF] LTND: Received terminated 2022-03-02 23:22:49.576 [INF] LTND: Shutting down... 2022-03-02 23:22:49.576 [INF] LTND: Gracefully shutting down. 2022-03-02 23:22:56.417 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:22:56.417 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:23:24.120 [INF] HLCK: Health check: chain backend, call: 1 failed with: health check: chain backend timed out after: 30s, backing off for: 2m0s 2022-03-02 23:23:33.667 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:23:33.667 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:23:41.727 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:23:41.727 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:23:49.795 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:23:49.795 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:23:57.856 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:23:57.856 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:05.923 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:05.923 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:13.987 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:13.988 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:22.051 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:22.051 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:30.111 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:30.111 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:38.179 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:38.179 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:45.343 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:45.343 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:24:53.407 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:24:53.407 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:00.447 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:00.447 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:08.510 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:08.510 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:15.552 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:15.552 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:23.616 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:23.616 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:25.655 [INF] HLCK: Health check: chain backend, call: 2 failed with: Post "http://10.21.21.8:8332": dial tcp 10.21.21.8:8332: connect: no route to host, backing off for: 2m0s 2022-03-02 23:25:30.655 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:30.655 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:38.719 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:38.719 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:46.783 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:46.783 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:25:54.848 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:25:54.849 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:02.915 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:02.915 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:10.979 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:10.979 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:19.042 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:19.043 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:27.107 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:27.108 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:35.167 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:35.167 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:43.233 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:43.234 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:51.296 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:51.296 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:26:59.359 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:26:59.359 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:07.426 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:07.426 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:15.491 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:15.491 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:23.554 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:23.555 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:26.615 [CRT] SRVR: Health check: chain backend failed after 3 calls 2022-03-02 23:27:26.615 [INF] SRVR: Sending request for shutdown 2022-03-02 23:27:26.615 [DBG] HLCK: Health check: max attempts failed, monitor exiting 2022-03-02 23:27:31.618 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:31.618 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:39.683 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:39.683 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:47.743 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:47.743 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:27:55.807 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:27:55.807 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:28:03.875 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:28:03.876 [TRC] LNWL: Re-establishing timed out ZMQ block connection 2022-03-02 23:28:11.935 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection 2022-03-02 23:28:11.936 [TRC] LNWL: Re-establishing timed out ZMQ block connection

AZBTC21 commented 2 years ago

Yes, the force closures have a transaction ID and I can see them on the blockchain at mempool.space. Ok, that puts me at ease that I will at least get these returned. I might just not be technical enough to follow your instructions as detailed as we both would prefer. My apologies if that is the case. I am certainly up to just force closing all my channels at this point. Yes, I can see my channels.db via SSH in the following location below. Can you help me on how to force close these all using the channels.db? Thanks so much.

umbrel@umbrel:~/umbrel/lnd/data/graph/mainnet $ ls -l total 4533328 -rw------- 1 umbrel umbrel 4311764992 Mar 3 01:55 channel.db

AZBTC21 commented 2 years ago

Can I just run ~/umbrel/bin/lncli closeallchannels --force to force close all? Will this use my channel.db? Thanks

lucasdcf commented 2 years ago

Oh, looks like we found something there. I think this could be your issue 2022-03-02 23:28:11.935 [TRC] LNWL: Re-establishing timed out ZMQ transaction connection

can you share your bitcoin.conf and your lnd.conf so we can check your config?

AZBTC21 commented 2 years ago

Hi Lucas - Happy to. Please see below. Removed a few sensitive values to keep private. Anything you see strange?

umbrel@umbrel:~/umbrel/bitcoin $ cat bitcoin.conf

Warning: it's not recommended to modify these files directly. Any

modifications you make can break the functionality of your umbrel. These files

are automatically reset with every Umbrel update.

Tor

proxy=xxxxxxxx listen=1 bind=xxxxxxxxx

Connections

port=8333 rpcport=8332 rpcbind=xxxxxxxx rpcbind=xxxxxxxx rpcallowip=xxxxxxxx rpcallowip=xxxxxxxx rpcauth=umbrel:03462e71c1ce50ed0b473920afa8exxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rescan=1

Memory

dbcache=200 maxmempool=300

zmq

zmqpubrawblock=tcp://0.0.0.0:28332 zmqpubrawtx=tcp://0.0.0.0:28333 zmqpubhashblock=tcp://0.0.0.0:28334 zmqpubsequence=tcp://0.0.0.0:28335

Indexes

txindex=1 blockfilterindex=1

SPV

peerbloomfilters=1 peerblockfilters=1

Compatibility

deprecatedrpc=addresses # https://github.com/mempool/mempool/issues/778#issuecomment-919194258 rpcworkqueue=64 # https://code.samourai.io/dojo/samourai-dojo/-/commit/54836226af024cb880ab490c994a9d7b0cd7afad

umbrel@umbrel:~/umbrel/lnd $ cat lnd.conf

Warning: it's not recommended to modify these files directly. Any

modifications you make can break the functionality of your umbrel.

[Application Options] listen=0.0.0.0:9735 rpclisten=0.0.0.0:10009 restlisten=0.0.0.0:8080 maxpendingchannels=3 minchansize=1000000 accept-keysend=true tlsextraip=xxxxxxxxxxxx tlsextradomain=xxxxxxxxxxx tlsautorefresh=1 tlsdisableautofill=1 feeurl= alias=AZBTC21 bitcoin.feerate=350 bitcoin.basefee=1 protocol.wumbo-channels=1

[Bitcoind] bitcoind.rpchost=10.21.21.8 bitcoind.rpcuser=umbrel bitcoind.rpcpass=xxxxxxxxxxxxxxxxxxxxx bitcoind.zmqpubrawblock=tcp://xxxxxxxxxxxxx bitcoind.zmqpubrawtx=tcp://xxxxxxxxxxxxxxx

[Bitcoin] bitcoin.active=1 bitcoin.mainnet=1

Default to neutrino as the node is

automatically switched to bitcoind once

IBD is complete

bitcoin.node=bitcoind bitcoin.defaultchanconfs=2

[neutrino]

Testnet neutrino peers that are automatically

uncommented if Umbrel is configured for testnet

neutrino.addpeer=testnet1-btcd.zaphq.io

neutrino.addpeer=testnet2-btcd.zaphq.io

[tor] tor.active=1 tor.control=xxxxxxxxxxxxxx tor.socks=xxxxxxxxxxxxxxx tor.targetipaddress=xxxxxxxxxxxxxx tor.password=axxxxxxxxxxxxxxxx tor.v3=1

arshbot commented 2 years ago

@AZBTC21 can you provide the lnd.conf file as well? Thinking lnd isn't connecting properly to the zmq socket

AZBTC21 commented 2 years ago

The lnd.conf is the last section after above after 'umbrel@umbrel:~/umbrel/lnd $ cat lnd.conf'

Sorry for the formatting in bold. The website automatically did that...

Thank you.

arshbot commented 2 years ago

@AZBTC21 Can you try setting the following values in lnd.conf?

bitcoind.zmqpubrawblock=tcp://0.0.0.0:28332
bitcoind.zmqpubrawtx=tcp://0.0.0.0:28333

You redacted the values in lnd.conf, so I'm wondering if they were set to an external ip address. They should be hooked into the corresponding bitcoind ports specified in your config

AZBTC21 commented 2 years ago

Yes, in the lnd.conf the values do have a value and are not 0's as they are in the bitcoin.conf file.

Will make the updates above in my lnd.conf file and reboot.

AZBTC21 commented 2 years ago

It does not seem to have helped. When looking at the log it seems like the wallet is locked, then I see it unlock briefly, before the zmq connection refusal. Then it locks back up and reunlocks. Here is some detail from the log:

2022-03-03 16:17:04.649 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:06.106 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:09.530 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:09.589 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:12.841 [INF] LNWL: Opened wallet 2022-03-03 16:17:13.115 [INF] CHRE: Primary chain is set to: bitcoin 2022-03-03 16:17:13.154 [ERR] LTND: unable to create partial chain control: unable to subscribe for zmq block events: dial tcp 0.0.0.0:28332: connect: connection refused 2022-03-03 16:17:13.155 [ERR] LTND: Shutting down because error in main method: error creating wallet config: unable to create partial chain control: unable to subscribe for zmq block events: dial tcp 0.0.0.0:28332: connect: connection refused 2022-03-03 16:17:13.649 [INF] LTND: Shutdown complete2022-03-03 16:17:15.264 [INF] LTND: Version: 0.14.2-beta commit=v0.14.2-beta, build=production, logging=default, debuglevel=info 2022-03-03 16:17:15.264 [INF] LTND: Active chain: Bitcoin (network=mainnet) 2022-03-03 16:17:15.266 [INF] RPCS: RPC server listening on 0.0.0.0:10009 2022-03-03 16:17:15.270 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080 2022-03-03 16:17:15.271 [INF] LTND: Opening the main database, this might take a few minutes... 2022-03-03 16:17:15.271 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false 2022-03-03 16:17:15.317 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available 2022-03-03 16:17:15.319 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available 2022-03-03 16:17:21.117 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available 2022-03-03 16:17:24.140 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available 2022-03-03 16:17:24.166 [ERR] RPCS: [/lnrpc.Lightning/GetTransactions]: waiting to start, RPC services not available 2022-03-03 16:17:24.218 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available 2022-03-03 16:17:24.219 [ERR] RPCS: [/lnrpc.Lightning/ListInvoices]: waiting to start, RPC services not available 2022-03-03 16:17:24.222 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available 2022-03-03 16:17:24.237 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available 2022-03-03 16:17:24.364 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: waiting to start, RPC services not available 2022-03-03 16:17:24.365 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available 2022-03-03 16:17:24.366 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available 2022-03-03 16:17:24.366 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available 2022-03-03 16:17:24.367 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available 2022-03-03 16:17:24.391 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: waiting to start, RPC services not available 2022-03-03 16:17:24.443 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: waiting to start, RPC services not available 2022-03-03 16:17:24.445 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: waiting to start, RPC services not available 2022-03-03 16:17:26.718 [INF] LTND: Creating local graph and channel state DB instances 2022-03-03 16:17:30.012 [INF] CHDB: Checking for schema update: latest_version=24, db_version=24 2022-03-03 16:17:30.012 [INF] LTND: Database(s) now open (time_to_open=14.740756353s)! 2022-03-03 16:17:30.013 [INF] LTND: We're not running within systemd 2022-03-03 16:17:30.013 [INF] LTND: Waiting for wallet encryption password. Use lncli create to create a wallet, lncli unlock to unlock an existing wallet, or lncli changepassword to change the password of an existing wallet and unlock it. 2022-03-03 16:17:33.997 [ERR] RPCS: [/lnrpc.Lightning/SubscribeInvoices]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:33.997 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:33.997 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:33.997 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.340 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.340 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.341 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.341 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.343 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.401 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:34.402 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:36.126 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.797 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.802 [ERR] RPCS: [/lnrpc.Lightning/ListInvoices]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.803 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.803 [ERR] RPCS: [/lnrpc.Lightning/GetTransactions]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.804 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.804 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.961 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.961 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.962 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.962 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.965 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:44.991 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:45.038 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:45.038 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:51.135 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.033 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.033 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.036 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.036 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.037 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.152 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:54.153 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:17:59.504 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.132 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.145 [ERR] RPCS: [/lnrpc.Lightning/GetTransactions]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.152 [ERR] RPCS: [/lnrpc.Lightning/ListInvoices]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.164 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.219 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.220 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:03.271 [ERR] RPCS: [/lnrpc.Lightning/ListPayments]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.566 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.567 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.568 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.579 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.585 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.607 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.736 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:05.737 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:06.152 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:13.142 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:13.211 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.116 [ERR] RPCS: [/lnrpc.Lightning/ChannelBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.116 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.118 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.119 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.122 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.183 [ERR] RPCS: [/lnrpc.Lightning/ListChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:15.185 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: wallet locked, unlock it to enable full RPC access 2022-03-03 16:18:16.215 [INF] LNWL: Opened wallet 2022-03-03 16:18:16.511 [INF] CHRE: Primary chain is set to: bitcoin 2022-03-03 16:18:16.550 [ERR] LTND: unable to create partial chain control: unable to subscribe for zmq block events: dial tcp 0.0.0.0:28332: connect: connection refused 2022-03-03 16:18:16.550 [ERR] LTND: Shutting down because error in main method: error creating wallet config: unable to create partial chain control: unable to subscribe for zmq block events: dial tcp 0.0.0.0:28332: connect: connection refused 2022-03-03 16:18:16.844 [INF] LTND: Shutdown complete2022-03-03 16:18:18.459 [INF] LTND: Version: 0.14.2-beta commit=v0.14.2-beta, build=production, logging=default, debuglevel=info 2022-03-03 16:18:18.459 [INF] LTND: Active chain: Bitcoin (network=mainnet) 2022-03-03 16:18:18.462 [INF] RPCS: RPC server listening on 0.0.0.0:10009 2022-03-03 16:18:18.466 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080 2022-03-03 16:18:18.467 [INF] LTND: Opening the main database, this might take a few minutes... 2022-03-03 16:18:18.467 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false 2022-03-03 16:18:21.175 [ERR] RPCS: [/lnrpc.Lightning/GetInfo]: waiting to start, RPC services not available 2022-03-03 16:18:23.159 [ERR] RPCS: [/lnrpc.Lightning/WalletBalance]: waiting to start, RPC services not available

AZBTC21 commented 2 years ago

This is what is in the lnd.conf file right now for zmq: bitcoind.zmqpubrawblock=tcp://0.0.0.0:28332 bitcoind.zmqpubrawtx=tcp://0.0.0.0:28333

This is what is in the bitcoin.conf file right now for zmq: zmqpubrawblock=tcp://0.0.0.0:28332 zmqpubrawtx=tcp://0.0.0.0:28333

The lnd.conf previously had some values before you requested a update it to 0's. However, the bitcoin.conf file seems to have always been this way with 0 values.

AZBTC21 commented 2 years ago

Should I take the tcp values which were previously in the lnd.conf and update them to replace them in the bitcoin.conf? Essentially, synching them up with a value so they are not all 0's?

lucasdcf commented 2 years ago

Hey AZ, try that please.

btcschellingpt commented 2 years ago

@AZBTC21 and @lucasdcf : looks very similar to this issue https://github.com/mynodebtc/mynode/issues/619

Had to recover the node, now dealing with the zombie channels left over from nodes who were inactive at recovery time. Looks like this is definitely an LND issue and not related to Umbrel/MyNode deployment.

lucasdcf commented 2 years ago

@AZBTC21 Could you send me not the tail logs, but the full logs after restart?

AZBTC21 commented 2 years ago

Apologies Lucas. Had to move forward with restoring my umbrel node with a full fresh start using SCB. The restore was successful and funds fully retrieved. Appreciate all your help.