Closed vodolazzka closed 4 years ago
jamesdanielv
commented
4 years ago my guess is use t-2 instead of t-3 to keep the memory use lower. i would guess that functions data used are not calculated as mem used because when function is complete the stack is free'd up. each thread requires more memory for calls. mem usage can go above the 32 bit address space as well, so it might need to be compiled to 64 bit to use more memory and use more threads.
gcarq
commented
4 years ago As of now I computed the longest chain manually because the assumption was that the blocks (in .blk files) are stored in the correct order, which is not the case. So all blocks are held in memory until the correct next block is found. This wasn't a huge problem a few years ago, but now it is. This might be a side effect of this, or worse you can run out of memory as @jamesdanielv pointed out.
I'm currently working on utilizing the block index computed by Bitcoin Core which should fix this issue. The PR is still WIP, but might be merged the next days: https://github.com/gcarq/rusty-blockparser/pull/58
Ubuntu 18.04, built from sources and installed from repo. 470597 blocks, then quit. There is 594248 blocks in the blockchain. Thoughts?
{ "chain": "main", "blocks": 594248, "headers": 594248, "bestblockhash": "0000000000000000000fe03dc34486f74b05bfa7803d5a3ea6d3c363cda37e60", "difficulty": 10771996663680.4, "mediantime": 1568151710, "verificationprogress": 0.9999961678274221, "initialblockdownload": false, "chainwork": "00000000000000000000000000000000000000000865da7e399526255f6520d3", "size_on_disk": 271704376342, "pruned": false,[18:56:47] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:56:52] DEBUG - worker-0: Parsing blk01758.dat (132.95 Mb) [18:56:56] DEBUG - worker-0: Parsing blk01759.dat (133.64 Mb) [18:56:57] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:01] DEBUG - worker-0: Parsing blk01760.dat (133.41 Mb) [18:57:06] DEBUG - worker-0: Parsing blk01761.dat (134.14 Mb) [18:57:07] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:12] DEBUG - worker-0: Parsing blk01762.dat (134.06 Mb) [18:57:17] DEBUG - worker-0: Parsing blk01763.dat (133.97 Mb) [18:57:17] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 63 blocks/sec) [18:57:22] DEBUG - worker-0: Parsing blk01764.dat (133.70 Mb) [18:57:27] DEBUG - worker-0: Parsing blk01765.dat (133.34 Mb) [18:57:27] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:32] DEBUG - worker-0: Parsing blk01766.dat (132.97 Mb) [18:57:37] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:38] DEBUG - worker-0: Parsing blk01767.dat (133.78 Mb) [18:57:43] DEBUG - worker-0: Parsing blk01768.dat (133.53 Mb) [18:57:47] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:48] DEBUG - worker-0: Parsing blk01769.dat (133.09 Mb) [18:57:52] DEBUG - worker-0: Parsing blk01770.dat (133.04 Mb) [18:57:57] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:57:58] DEBUG - worker-0: Parsing blk01771.dat (134.05 Mb) [18:58:03] DEBUG - worker-0: Parsing blk01772.dat (133.63 Mb) [18:58:07] INFO - dispatch: Status: 470597 Blocks processed. (left: 0, avg: 62 blocks/sec) [18:58:07] DEBUG - worker-0: Parsing blk01773.dat (133.70 Mb) [18:58:12] DEBUG - worker-0: Parsing blk01774.dat (132.92 Mb) [18:58:13] DEBUG - worker-0: Got 0x00000000 as magic number. Finished. [18:58:13] DEBUG - dispatch: worker-0 completed [18:58:13] INFO - dispatch: All threads finished. [18:58:13] INFO - dispatch: Done. Processed 470597 blocks in 125.37 minutes. (avg: 62 blocks/sec)bitcoind version:{ "version": 180000, "subversion": "/Satoshi:0.18.0/", "protocolversion": 70015, "localservices": "000000000000040d", "localrelay": true, "timeoffset": -1, "networkactive": true, "connections": 16,[00:04:59] INFO - dispatch: All threads finished. [00:04:59] INFO - dispatch: Done. Processed 594248 blocks in 7.26 minutes. (avg: 1365 blocks/sec) [00:04:59] INFO - dispatch: Saving block headers as ./chain.json ... [00:05:00] DEBUG - chain: Longest chain: -> height: 470596 -> newest block: 00000000000000000081e7ef68029833ebf5db8c458948cc911cf6e2b8fa9a40 -> genesis block: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f [00:05:00] DEBUG - chain: Genesis hash is valid. [00:05:00] DEBUG - chain: Inserted 470597 new blocks ... [00:05:01] DEBUG - chain: Serialized 470597 hashes to ./chain.json. Latest processed block height: 0 ... (latest blk.dat index: 0) [00:05:01] DEBUG - main: Iteration 1 finished. [00:05:02] DEBUG - chain: Imported 470597 hashes from ./chain.json. Current block height: 0 ... (latest blk.dat index: 0)Thoughts?