Blocks/sec too low - Githubissues

heidenrei commented 3 years ago

I ran rusty-blockparser csvdump ~/dump/ and the number of blocks/sec started high but quickly got too low for the program to be able to finish in a reasonable time. Did I make a mistake somewhere or is this just how long it will take?

[0:49:16] INFO - parser: Status: 110972 Blocks processed. (left: 553502, avg: 11097 blocks/sec) [0:49:26] INFO - parser: Status: 124330 Blocks processed. (left: 540144, avg: 6216 blocks/sec) [0:49:36] INFO - parser: Status: 130806 Blocks processed. (left: 533668, avg: 4360 blocks/sec) [0:49:46] INFO - parser: Status: 135074 Blocks processed. (left: 529400, avg: 3376 blocks/sec) [0:49:56] INFO - parser: Status: 139093 Blocks processed. (left: 525381, avg: 2781 blocks/sec) ... [3:42:48] INFO - parser: Status: 444530 Blocks processed. (left: 219944, avg: 42 blocks/sec) [3:42:58] INFO - parser: Status: 444636 Blocks processed. (left: 219838, avg: 42 blocks/sec) [3:43:08] INFO - parser: Status: 444769 Blocks processed. (left: 219705, avg: 42 blocks/sec) [3:43:18] INFO - parser: Status: 444899 Blocks processed. (left: 219575, avg: 42 blocks/sec) [3:43:28] INFO - parser: Status: 445056 Blocks processed. (left: 219418, avg: 42 blocks/sec)

divra056 commented 3 years ago

I have got same perf issue.

[12:34:02] INFO - main: Starting rusty-blockparser v0.8.1 ... [12:34:02] INFO - index: Reading index from E:\software\bitcoin\blocks\index ... [12:34:05] INFO - index: Got longest chain with 674715 blocks ... [12:34:05] INFO - blkfile: Reading files from E:\software\bitcoin\blocks ... [12:34:08] INFO - parser: Parsing Bitcoin blockchain (range=0..) ... [12:34:08] INFO - callback: Using balances with dump folder: E:\software ... [12:34:18] INFO - parser: Status: 85391 Blocks processed. (left: 589324, avg: 8539 blocks/sec) [12:34:28] INFO - parser: Status: 118405 Blocks processed. (left: 556310, avg: 5920 blocks/sec) [12:34:38] INFO - parser: Status: 130318 Blocks processed. (left: 544397, avg: 4343 blocks/sec) [12:34:48] INFO - parser: Status: 135775 Blocks processed. (left: 538940, avg: 3394 blocks/sec) [12:34:58] INFO - parser: Status: 140834 Blocks processed. (left: 533881, avg: 2816 blocks/sec) [12:35:08] INFO - parser: Status: 146592 Blocks processed. (left: 528123, avg: 2443 blocks/sec) [12:35:18] INFO - parser: Status: 154594 Blocks processed. (left: 520121, avg: 2208 blocks/sec) [12:35:28] INFO - parser: Status: 162954 Blocks processed. (left: 511761, avg: 2036 blocks/sec) [12:35:38] INFO - parser: Status: 170054 Blocks processed. (left: 504661, avg: 1889 blocks/sec) [12:35:48] INFO - parser: Status: 176567 Blocks processed. (left: 498148, avg: 1765 blocks/sec) [12:35:58] INFO - parser: Status: 180599 Blocks processed. (left: 494116, avg: 1641 blocks/sec) [12:36:08] INFO - parser: Status: 182423 Blocks processed. (left: 492292, avg: 1520 blocks/sec) [12:36:18] INFO - parser: Status: 184297 Blocks processed. (left: 490418, avg: 1417 blocks/sec) [12:36:28] INFO - parser: Status: 185568 Blocks processed. (left: 489147, avg: 1325 blocks/sec) [12:36:38] INFO - parser: Status: 187926 Blocks processed. (left: 486789, avg: 1252 blocks/sec) [12:36:48] INFO - parser: Status: 190044 Blocks processed. (left: 484671, avg: 1187 blocks/sec) [12:36:58] INFO - parser: Status: 192098 Blocks processed. (left: 482617, avg: 1129 blocks/sec) [12:37:08] INFO - parser: Status: 194021 Blocks processed. (left: 480694, avg: 1077 blocks/sec) [12:37:18] INFO - parser: Status: 195803 Blocks processed. (left: 478912, avg: 1030 blocks/sec) [12:37:28] INFO - parser: Status: 197527 Blocks processed. (left: 477188, avg: 987 blocks/sec) [12:37:38] INFO - parser: Status: 199471 Blocks processed. (left: 475244, avg: 949 blocks/sec) [12:37:48] INFO - parser: Status: 201456 Blocks processed. (left: 473259, avg: 915 blocks/sec) [12:37:58] INFO - parser: Status: 203035 Blocks processed. (left: 471680, avg: 882 blocks/sec) [12:38:08] INFO - parser: Status: 204620 Blocks processed. (left: 470095, avg: 852 blocks/sec) [12:38:18] INFO - parser: Status: 206651 Blocks processed. (left: 468064, avg: 826 blocks/sec) [12:38:28] INFO - parser: Status: 208328 Blocks processed. (left: 466387, avg: 801 blocks/sec) [12:38:38] INFO - parser: Status: 209902 Blocks processed. (left: 464813, avg: 777 blocks/sec) [12:38:48] INFO - parser: Status: 211561 Blocks processed. (left: 463154, avg: 755 blocks/sec) [12:38:58] INFO - parser: Status: 212800 Blocks processed. (left: 461915, avg: 733 blocks/sec) [12:39:08] INFO - parser: Status: 214156 Blocks processed. (left: 460559, avg: 713 blocks/sec) [12:39:18] INFO - parser: Status: 215688 Blocks processed. (left: 459027, avg: 695 blocks/sec) [12:39:28] INFO - parser: Status: 216773 Blocks processed. (left: 457942, avg: 677 blocks/sec) [12:39:38] INFO - parser: Status: 217887 Blocks processed. (left: 456828, avg: 660 blocks/sec)

Congyuwang commented 3 years ago

Try this if you are familiar with rust, and are willing to write a few lines of code yourself.

Much better performance.

https://github.com/Congyuwang/Rusty-Bitcoin-Explorer

This is a rust project based the current repo.

It has several improvements and differences:

fast performance: tested on 4 core i7 and SSD, with 8GB RAM, taking 2.5 hours to iterate through all blocks (2021-October) with input addresses found, using a maximum of 4GB RAM;
multi-threading: concurrent iterator interface with sequential output, and can take full advantage of more powerful CPUS;
much smaller RAM footprint using on-disk UTXO cache (v1.2 and up) (about 4GB RAM is enough)
supports a varieties of block formats: you can find input addresses using Rusty-Bitcoin-Explorer;
library instead of binary crate: Rusty-Bitcoin-Explorer is a library, thus allowing more versatile usage than a binary crate;
configurable to in-memory UTXO cache (32 GB RAM) but faster performance using default-features=false;

gcarq commented 1 year ago

I ran rusty-blockparser csvdump ~/dump/ and the number of blocks/sec started high but quickly got too low for the program to be able to finish in a reasonable time. Did I make a mistake somewhere or is this just how long it will take?

Nothing wrong here, the blocks in the beginning don't contain as much transactions as blocks later in time. However I published 0.9.0 which has a few performance improvements. Closing this

gcarq / rusty-blockparser

Blocks/sec too low #70