search

getumbrel / umbrel

A beautiful home server OS for self-hosting with an app store. Buy a pre-built Umbrel Home with umbrelOS, or install on a Raspberry Pi or any x86 system.
https://umbrel.com
Other
7.31k stars 527 forks source link

Connection to electrs seems to be failing #681

Closed bezysoftware closed 1 year ago

bezysoftware commented 3 years ago

BTC RPC Explorer won't load, there seems to be an issue with electrs, see from the logs:

Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.315Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.316Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"})
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.318Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.320Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.322Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"})
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.323Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
Mar 27 12:17:33 umbrel umbrel-app btc-rpc-explorer_web_1[717]: 2021-03-27T12:17:33.325Z btcexp:electrumx Disconnected from ElectrumX @ 10.21.21.10:50001

From the electrs logs:

Mar 27 12:20:24 umbrel electrs[717]: TRACE - indexed "/data/.bitcoin/blocks/blk01352.dat": 0 rows
Mar 27 12:20:25 umbrel electrs[717]: TRACE - indexed "/data/.bitcoin/blocks/blk01353.dat": 0 rows
Mar 27 12:20:28 umbrel electrs[717]: TRACE - indexed "/data/.bitcoin/blocks/blk01354.dat": 0 rows
Mar 27 12:20:28 umbrel electrs[717]: TRACE - indexed "/data/.bitcoin/blocks/blk01355.dat": 0 rows
...

Does this mean that electrs is still in process of indexing and refusing connections? Isn't it suspicious that it indexes "0 rows"? Any guess on how long it will take?

blckbx commented 3 years ago

I can confirm that. In addition it seems electrs runs out of memory and gets killed, if I understand it correctly (seen in kern.log and syslog):

Mar 28 17:31:57 pi4 electrs[1018]: TRACE - downloading 100000 block headers Mar 28 17:32:19 pi4 electrs[1018]: TRACE - downloading 100000 block headers Mar 28 17:32:39 pi4 electrs[1018]: TRACE - downloading 76715 block headers Mar 28 17:32:59 pi4 electrs[1018]: DEBUG - applying 676715 new headers from height 0 Mar 28 17:33:03 pi4 kernel: [111488.043686] [ 133683] 1000 133683 637829 345334 2969600 0 0 electrs Mar 28 17:33:03 pi4 kernel: [111488.043708] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=electrs,p> Mar 28 17:33:03 pi4 kernel: [111488.043790] Out of memory: Killed process 133683 (electrs) total-vm:2551316kB, anon-rss:1381336kB, file-rss:0kB, shmem-rss:0kB> Mar 28 17:33:03 pi4 kernel: [111488.270495] oom_reaper: reaped process 133683 (electrs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

louneskmt commented 3 years ago

What hardware do you two use?

bezysoftware commented 3 years ago

Raspberry Pi 4B - 4 GB RAM; Umbrel v0.3.8. I haven't observed the issue today, it's possible it was just during initial load

blckbx commented 3 years ago
blckbx commented 3 years ago

I removed the apps mempool and btc_rpc_explorer because they tried to connect with electrs to see if this is of any help but to no avail. Watched the logs after app removal. Electrs gets killed after indexing failed (see electrs log posted by @bezysoftware). Attached is the log of the crash. This happens regularly every 4-5 Minutes as far as I can see. electrs_crash.log

AaronDewes commented 3 years ago

I removed the apps mempool and btc_rpc_explorer because they tried to connect with electrs to see if this is of any help but to no avail. Watched the logs after app removal. Electrs gets killed after indexing failed (see electrs log posted by @bezysoftware). Attached is the log of the crash. This happens regularly every 4-5 Minutes as far as I can see. electrs_crash.log

Can you try to remove all apps for now and set up a swapfile? electrs seems to be running out of RAM. We have a 4GB swapfile on Umbrel OS to avooid this issue, I'd recommend adding one for you to or directly using Umbrel OS.

blckbx commented 3 years ago

Ok, removed all apps, changed the swapfile from 1GB to 4GB and I'm currently reindexing the blockchain, just in case something went wrong on IBD.

blckbx commented 3 years ago

Short recap:

Apr 1 14:07:50 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00624.dat": 1511539 rows Apr 1 14:07:53 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00625.dat": 1517639 rows Apr 1 14:07:57 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00626.dat": 1476823 rows Apr 1 14:08:02 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00627.dat": 1553775 rows Apr 1 14:08:07 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00628.dat": 1535316 rows Apr 1 14:08:10 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00629.dat": 1608106 rows Apr 1 14:08:14 pi4 electrs[1023]: TRACE - indexed "/data/.bitcoin/blocks/blk00630.dat": 1563344 rows

blckbx commented 3 years ago

Update: eletcrs is done indexing, but "full compaction" seems to be ongoing. Once either mempool or btc_rpc_explorer is installed, connection errors are being thrown into syslog: Apr 1 17:36:30 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:30 [62] ERR: Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:36:30 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:30 [62] ERR: Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:36:30 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:30 [62] INFO: Disconnected from Electrum Server at 10.21.21.10:50001 Apr 1 17:36:40 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:40 [62] ERR: Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:36:40 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:40 [62] ERR: Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:36:40 pi4 umbrel-app mempool_api_1[1023]: Apr 1 15:36:40 [62] INFO: Disconnected from Electrum Server at 10.21.21.10:50001

Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.781Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.781Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"}) Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.782Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001 Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.782Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001} Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.783Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"}) Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.783Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001 Apr 1 17:42:15 pi4 umbrel-app btc-rpc-explorer_web_1[1023]: 2021-04-01T15:42:15.783Z btcexp:electrumx Disconnected from ElectrumX @ 10.21.21.10:50001

Any idea what's going on here? electrs binds to 50001 just fine: getumbrel/electrs:v0.8.9 "electrs" 32 hours ago Up 32 hours 4224/tcp, 0.0.0.0:50001->50001/tcp

blckbx commented 3 years ago

Alright, I think I got it working. It seems to be a timing problem. After IBD/reindexing electrs needs time to do some more work (full compaction?). After that I can install mempool just fine. Mempool itself needs time to fetch all transactions (history) of the mempool before you can interact with the gui. Keeping fingers crossed: Mempool, BTC_RPC_Explorer, Thunderhub and RTL are installed now with no errors so far. Maybe it's a good idea to keep explorer apps unavailable until IBD/reindex is done + 1day cleanup (compaction) time for electrs. Raspibolt writes: Wait until the initial sync is completed and the database is compacted. The space used on disk will grow to over 125 GB before reducing to about 60 GB at the time of this writing. (https://stadicus.github.io/RaspiBolt/raspibolt_50_electrs.html) It took about 12 hrs to complete compaction.

rafaelpac commented 3 years ago

I have a similar situation, going to leave here some data I have, but not sure I understand it correctly. I am running Umbrel on an Ubuntu machine, freshly started for it. After deploy, I stopped it and copied my blocks and chainstate fully syncd folders. Restarted it and saw that it was doing txindex and block filter index. Today in the morning the txindex finished like this below:

2021-04-04T06:46:01Z Syncing txindex with block chain from height 677540
2021-04-04T06:46:32Z Syncing txindex with block chain from height 677589
2021-04-04T06:47:03Z Syncing txindex with block chain from height 677686
2021-04-04T06:47:03Z txindex is enabled at height 677688
2021-04-04T06:47:03Z txindex thread exit
2021-04-04T06:49:49Z Pre-allocating up to position 0xd00000 in rev02522.dat
2021-04-04T06:49:49Z UpdateTip: new best=000000000000000000056919fb8f640c5e09f3a446063afd19f237909cdbcc93 height=677689 version=0x20004000 log2_work=92.784824 tx=630930041 date='2021-04-04T06:49:33Z' progress=1.000000 cache=166.2MiB(1223480txo)
2021-04-04T06:58:00Z UpdateTip: new best=0000000000000000000267cb58833f3ac53bd3a5d93883629c4f246f68677f83 height=677690 version=0x20000000 log2_work=92.784841 tx=630931868 date='2021-04-04T06:57:11Z' progress=1.000000 cache=166.6MiB(1227037txo)

My btc-rpc-explorer was giving me: Error: No ElectrumX connection available.

I tried to restart the system and now I get:

Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.063Z btcexp:electrumx Electrum attempting reconnect...
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.064Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"})
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:electrumx Electrum error: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:error Error 937gf47dsyde: Error: connect ECONNREFUSED 10.21.21.10:50001, json: {"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"10.21.21.10","port":50001}, userData: [object Object] (json: {"host":"10.21.21.10","port":50001,"protocol":"tcp"})
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED 10.21.21.10:50001
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
Apr  4 12:19:30 rafaelpac-umbrel umbrel-app btc-rpc-explorer_web_1[1396]: 2021-04-04T15:19:30.065Z btcexp:electrumx Disconnected from ElectrumX @ 10.21.21.10:50001

The solution then is to reindex? Or the wait for compaction should be enough?

nmfretz commented 1 year ago

closing old support issue