search

romanz / electrs

An efficient re-implementation of Electrum Server in Rust
MIT License
1.09k stars 403 forks source link

Bug: Unable to sync, failed to get block #1071

Open asyscom opened 2 months ago

asyscom commented 2 months ago

Hello, I can’t sync electrs. Every time I launch the program, after a variable time between 10 minutes and an hour, the sync gets stuck and I receive this error:

Error: electrs failed

Caused by:
    0: sync failed
    1: failed to get block 00000000000000013c7b5580791075c98fa0bd2506ed777d9362033b49790cf5
    2: receiving on an empty and disconnected channel

If I restart the Bitcoin service and then electrs resumes syncing, but after a while, it gets stuck again. Obviously, I can't keep restarting Bitcoin continuously. The node works correctly, including integration with LND, ThunderHub, and LIT.

Electrs version Electrs version 0.10.5

System running electrs SO: Ubunu Server 22 Bitcoin: version v27.1.0 LND: 0.18.2

Thanks in advance Davide

romanz commented 2 months ago

Please make sure your bitcoind is configured with the following flags: https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
whitelist=download@127.0.0.1
asyscom commented 2 months ago

Please make sure your bitcoind is configured with the following flags: https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
whitelist=download@127.0.0.1

Thanks, i've update my bitcoin,conf

datadir=/var/lib/bitcoind

bitcoind configuration

some values might be overruled directly systemd-service exec call parameters

mainnet/testnet

testnet=0

Bitcoind options

server=1 daemon=1 txindex=1 disablewallet=0 peerbloomfilters=1 datadir=/var/lib/bitcoind main.debuglogfile=/var/lib/bitcoind/debug.log

Connection settings

rpcuser=xxxxxx rpcpassword=xxxxxxx main.rpcport=8332 test.rpcport=18332 rpcallowip=127.0.0.1 main.rpcbind=127.0.0.1:8332 test.rpcbind=127.0.0.1:18332 main.zmqpubrawblock=tcp://127.0.0.1:28332 main.zmqpubrawtx=tcp://127.0.0.1:28333

Raspberry Contabo

dbcache=2048 maxorphantx=10 maxmempool=500

maxconnections=400

maxuploadtarget=5000 onlynet=ipv4 onlynet=onion

Tor

proxy=127.0.0.1:9050 listen=1 listenonion=1 bind=127.0.0.1 externalip=my-onio

custom

externalip=my-ip maxconnections=N whitelist=download@127.0.0.1

After I restarted bitcoin, lne and electr the sync resumed for a 10 minutes and then gave the same error again

Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 } [2024-08-16T09:44:04.582Z DEBUG tiny_http] Server listening on 127.0.0.1:4224 [2024-08-16T09:44:04.582Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224 [2024-08-16T09:44:04.582Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001 [2024-08-16T09:44:04.582Z DEBUG tiny_http] Running accept thread [2024-08-16T09:44:04.611Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 76 SST files, 0.63497689 GB, 0.048874206 Grows [2024-08-16T09:44:04.611Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 }) [2024-08-16T09:44:04.908Z INFO electrs::chain] loading 214350 headers, tip=000000000000019b729b686a16de4c46da980c9fd8709043de36b39c1dee820a [2024-08-16T09:44:05.165Z INFO electrs::chain] chain updated: tip=000000000000019b729b686a16de4c46da980c9fd8709043de36b39c1dee820a, height=214350 [2024-08-16T09:44:05.168Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-16T09:44:05.169Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T09:44:05.169Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T09:44:05.170Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-16T09:44:05.171Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T09:44:05.175Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723801445, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 2267302671146926227, user_agent: "/Satoshi:27.1.0/", start_height: 857018, relay: true } [2024-08-16T09:44:05.178Z DEBUG electrs::p2p] got 2000 new headers [2024-08-16T09:44:05.179Z INFO electrs::index] indexing 2000 blocks: [214351..216350] [2024-08-16T09:44:05.179Z DEBUG electrs::p2p] loading 10 blocks [2024-08-16T09:44:05.230Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed [2024-08-16T09:44:05.232Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected [2024-08-16T09:44:05.232Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin [2024-08-16T09:44:05.232Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send Error: electrs failed

Caused by: 0: sync failed 1: failed to get block 0000000000000247f6cc4f073a9b38a770566d51e03bdb588cf1b7fa5ac41782 2: receiving on an empty and disconnected channel

asyscom commented 2 months ago

Please make sure your bitcoind is configured with the following flags: https://github.com/romanz/electrs/blob/master/doc/config.md#bitcoind-configuration

maxconnections=N
whitelist=download@127.0.0.1

ive triedto remove maxuploadtarget and restart all services but after some minute riceived the same error Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 } [2024-08-16T11:03:40.580Z DEBUG tiny_http] Server listening on 127.0.0.1:4224 [2024-08-16T11:03:40.580Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224 [2024-08-16T11:03:40.580Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001 [2024-08-16T11:03:40.581Z DEBUG tiny_http] Running accept thread [2024-08-16T11:03:40.607Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 326 SST files, 1.216418774 GB, 0.096343168 Grows [2024-08-16T11:03:40.607Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 }) [2024-08-16T11:03:41.002Z INFO electrs::chain] loading 245120 headers, tip=00000000000000672b757cf1acd624a0a9d5bef7b641929b5e270ad8fd9b4af8 [2024-08-16T11:03:41.317Z INFO electrs::chain] chain updated: tip=00000000000000672b757cf1acd624a0a9d5bef7b641929b5e270ad8fd9b4af8, height=245120 [2024-08-16T11:03:41.325Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-16T11:03:41.326Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T11:03:41.326Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T11:03:41.327Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-16T11:03:41.327Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-16T11:03:41.329Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723806221, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 14602067919111189498, user_agent: "/Satoshi:27.1.0/", start_height: 857029, relay: true } [2024-08-16T11:03:41.334Z DEBUG electrs::p2p] got 2000 new headers [2024-08-16T11:03:41.335Z INFO electrs::index] indexing 2000 blocks: [245121..247120] [2024-08-16T11:03:41.335Z DEBUG electrs::p2p] loading 10 blocks [2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed [2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected [2024-08-16T11:03:41.386Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send [2024-08-16T11:03:41.386Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin Error: electrs failed

Caused by: 0: sync failed 1: failed to get block 0000000000000050b60fecadb2146fd594db80b5bca34ec03862d3caa2a53523 2: receiving on an empty and disconnected channel

Kixunil commented 2 months ago

I assume maxconnections shouldn't be lieral N but some high number (100?).

asyscom commented 2 months ago

I assume maxconnections shouldn't be lieral N but some high number (100?).

Hum I meant that I can put N because it's for infinite. Need i put a number?

Kixunil commented 2 months ago

Check bitcoind's doc.

asyscom commented 2 months ago

Check bitcoind's doc.

OK, tomorrow I'll set it to 200. Let me time to try and give a feedback

asyscom commented 2 months ago

Check bitcoind's doc.

nothin to do, i'se to 400 maxconnections, removemaxuploadtarget, set whitelist=download@127.0.0.1 but the errors is still the same Starting electrs 0.10.5 on x86_64 linux with Config { network: Bitcoin, db_path: "/opt/data/electrs/.electrs/db/bitcoin", db_log_dir: None, daemon_dir: "/usr/bin/bitcoind", daemon_auth: UserPass("bitcoind", ""), daemon_rpc_addr: 127.0.0.1:8332, daemon_p2p_addr: 0.0.0.0:8333, electrum_rpc_addr: 0.0.0.0:50001, monitoring_addr: 127.0.0.1:4224, wait_duration: 10s, jsonrpc_timeout: 15s, index_batch_size: 10, index_lookup_limit: Some(1000), reindex_last_blocks: 0, auto_reindex: true, ignore_mempool: false, sync_once: false, skip_block_download_wait: false, disable_electrum_rpc: false, server_banner: "Welcome to electrs 0.10.5 (Electrum Rust Server)!", signet_magic: f9beb4d9 } [2024-08-18T10:06:47.126Z DEBUG tiny_http] Server listening on 127.0.0.1:4224 [2024-08-18T10:06:47.127Z INFO electrs::metrics::metrics_impl] serving Prometheus metrics on 127.0.0.1:4224 [2024-08-18T10:06:47.127Z INFO electrs::server] serving Electrum RPC on 0.0.0.0:50001 [2024-08-18T10:06:47.127Z DEBUG tiny_http] Running accept thread [2024-08-18T10:06:47.156Z INFO electrs::db] "/opt/data/electrs/.electrs/db/bitcoin": 363 SST files, 1.818754212 GB, 0.144986034 Grows [2024-08-18T10:06:47.157Z DEBUG electrs::db] DB Some(Config { compacted: false, format: 0 }) [2024-08-18T10:06:47.642Z INFO electrs::chain] loading 275880 headers, tip=0000000000000003938261a92c469e8474754d6a23b1e5ed1219159c50b0a200 [2024-08-18T10:06:47.999Z INFO electrs::chain] chain updated: tip=0000000000000003938261a92c469e8474754d6a23b1e5ed1219159c50b0a200, height=275880 [2024-08-18T10:06:48.007Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-18T10:06:48.007Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-18T10:06:48.008Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-18T10:06:48.009Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo [] [2024-08-18T10:06:48.009Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo [] [2024-08-18T10:06:48.012Z DEBUG electrs::p2p] peer version: VersionMessage { version: 70016, services: ServiceFlags(3085), timestamp: 1723975608, receiver: Address {services: ServiceFlags(NONE), address: 0.0.0.0, port: 0}, sender: Address {services: ServiceFlags(NETWORK|BLOOM|WITNESS|NETWORK_LIMITED|P2P_V2), address: 0.0.0.0, port: 0}, nonce: 4227069755078880485, user_agent: "/Satoshi:27.1.0/", start_height: 857324, relay: true } [2024-08-18T10:06:48.015Z DEBUG electrs::p2p] got 2000 new headers [2024-08-18T10:06:48.016Z INFO electrs::index] indexing 2000 blocks: [275881..277880] [2024-08-18T10:06:48.016Z DEBUG electrs::p2p] loading 10 blocks [2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_recv thread: connection closed [2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_loop thread: peer has disconnected [2024-08-18T10:06:48.067Z DEBUG electrs::p2p] closing p2p_send thread: no more messages to send [2024-08-18T10:06:48.067Z INFO electrs::db] closing DB at /opt/data/electrs/.electrs/db/bitcoin Error: electrs failed

Caused by: 0: sync failed 1: failed to get block 00000000000000000785bdd27a61fdd9c82b033871cf544b2a77cda0d0ff321e 2: receiving on an empty and disconnected channel

and electrs never restart until restart bitcoind I've tried to delete /opt/data/electrs/.electrs/db/bitcoin content without lucky

asyscom commented 2 months ago

Check bitcoind's doc.

The last attempt was to uninstall and reinstall Electrs, but it still doesn't start. I have to restart Bitcoin Core to get it running again, but after just a few minutes of syncing, the problem reoccurs. Any suggestions?

antonilol commented 2 months ago

How is ram, cpu and disk usage just before the error occurs?

asyscom commented 2 months ago

How is ram, cpu and disk usage just before the error occurs?

Hello, I've 27gb of ram, 500gb of free disk on total of 2tb and 8 cpu

Kixunil commented 2 months ago

Any logs in bitcoind around that time?

antonilol commented 2 months ago

Hello, I've 27gb of ram, 500gb of free disk on total of 2tb and 8 cpu

Assuming that's 27 GB free (not total) that should be more than enough, I have had issues with bitcoind rpc timing out on close to 100% memory usage or prolonged 100% cpu usage. 500 GB free disk is also more than enough. My electrs mainnet database is currently around 48 GB.