search

Blockstream / esplora

Explorer for Bitcoin and Liquid
MIT License
1.02k stars 400 forks source link

Extremely slow indexing on good hardware #536

Open pokrovskyy opened 1 month ago

pokrovskyy commented 1 month ago

How long is it even supposed to take to do the lightmode index? I'm running a server with 2.5TB datacenter-grade SSD RAID, 64GB RAM, Intel Core i7-8700. Had Bitcoind already synced. It took probably 12hrs to index and compact txstore DB, and now it's 5th hour of history indexing and it seems to go waaay slower (currently on 300 of 4.8k .blk files) Not sure I understand what's the bottleneck here and the timeframe for the sync? CPU usage is <15%, mem 10%. Please advise!

philippem commented 3 weeks ago

Initial sync is a time consuming and heavy operation, but I'd expect the CPU usage to be higher. It may take several days. How are you invoking the esplora? are you running it in a single docker container as in the README?

pokrovskyy commented 3 weeks ago

@philippem I was running it directly from bash:

cargo run --release --bin electrs -- -vvvv --daemon-dir ~/.bitcoin --lightmode

rustc 1.82.0 (f6e511eec 2024-10-15) cargo 1.82.0 (8f40fc59f 2024-08-21)

philippem commented 3 weeks ago

It will take several days for initial indexing and database compaction. Memory usage may go up to 15-20 GB then drop. WIth the -vvvv option there will be trace and debug messages which indicating progress. The CPU usage will rise and fall depending on which phase of the sync it is doing, and the speed of the filesystem.

Electrs can be stopped and restarted and it will resume where it left off. Once the initial sync is complete, subsequent starts will be much faster (only newer blocks be indexed).