Error Magic Number

[Nellous]

Hello,

Im using your program to parse the bitcoin blockchain, everything goes smooth until i get this error. Im not able to identify which is the exact block causing the problem, but probably is located between blk00308.dat and blk00311.dat

[15:44:25] INFO - dispatch: Status: 367219 Blocks added to index. (avg: 307 blocks/sec) [15:44:25] DEBUG - worker-3: Got 0x00000000 as magic number. Finished. [15:44:25] DEBUG - dispatch: worker-3 completed [15:44:25] ERROR - worker-2: Got invalid magic value for Bitcoin: 0x36b2e95, expected: 0xd9b4bef9 Validation Error [15:44:25] ERROR - dispatch: Got invalid magic value for Bitcoin: 0x36b2e95, expected: 0xd9b4bef9 Validation Error

Is it a program configuration related error or is my blockchain corrupted?

Any help would be greatly appreciated!

And thanks for your great program!

[gcarq]

Which bitcoin client version are you using?

It seems this is not related to a configuration, but rather a filesystem corruption, a bug or a change in the latest bitcoin core release.

I will try to reproduce it

[Nellous]

Im using bitcoin core 0.14.0

I re-downloaded the affected part of the blockchain, and im testing right now to see if the problem persists.

Thanks for the fast reply!

[gcarq]

I cannot reproduce it on my machine:

[18:50:01] INFO - dispatch: Status: 453823 Blocks added to index. (avg:   564 blocks/sec)
[18:50:11] INFO - dispatch: Status: 455188 Blocks added to index. (avg:   559 blocks/sec)
[18:50:21] INFO - dispatch: Status: 456527 Blocks added to index. (avg:   554 blocks/sec)
[18:50:31] INFO - dispatch: Status: 457859 Blocks added to index. (avg:   548 blocks/sec)
[18:50:41] INFO - dispatch: Status: 459270 Blocks added to index. (avg:   544 blocks/sec)
[18:50:51] INFO - dispatch: Status: 460632 Blocks added to index. (avg:   539 blocks/sec)
[18:50:58] INFO - dispatch: All threads finished.
[18:50:58] INFO - dispatch: Done. Processed 461571 blocks in 14.36 minutes. (avg:   536 blocks/sec)
[18:50:58] INFO - dispatch: Saving block headers as chain.json ...

[Nellous]

Thanks you very much for your support. You helped me understand what is (probably) the problem.

It probably lies in the fact that the blockchain files i was trying to extract with your script, were downloaded using different machines with different OSs. In fact, i tested re-downloading only the error-affected and subsequent .blk files and resulted in this error.

[17:52:30] DEBUG - worker-1: Parsing blk00300.dat (134.17 Mb)
[17:52:31] INFO - dispatch: Status: 364742 Blocks processed. (left:    952, avg:    69 blocks/sec)
[17:52:38] DEBUG - worker-0: Parsing blk00301.dat (133.57 Mb)
[17:52:41] INFO - dispatch: Status: 364742 Blocks processed. (left:    952, avg:    69 blocks/sec)
[17:52:51] INFO - dispatch: Status: 364750 Blocks processed. (left:    944, avg:    68 blocks/sec)
[17:52:56] DEBUG - worker-1: Parsing blk00302.dat (133.29 Mb)
[17:53:01] INFO - dispatch: Status: 364807 Blocks processed. (left:    887, avg:    68 blocks/sec)
[17:53:03] DEBUG - worker-0: Parsing blk00303.dat (133.83 Mb)
[17:53:11] INFO - dispatch: Status: 364891 Blocks processed. (left:    803, avg:    68 blocks/sec)
[17:53:19] ERROR - worker-0: I/O Error: failed to fill whole buffer
[17:53:19] ERROR - dispatch: I/O Error: failed to fill whole buffer

thread 'worker-1' has overflowed its stack
fatal runtime error: stack overflow
Abort trap: 6

It took about 2 hours to index the first 364K blocks :/ (Don't know if the slowness is hardware related or related to the probably corrupted .blk files ). Is amazing how fast is the indexing performed on your machine.

I am now re-downloading the whole BlockChain from a single machine and without stopping and resuming the sync.

Once the download is completed, i will re-launch your script and let you know the results..

In the mean time, if it's not too much bother, could you share your csvdump (through an ftp server or in the way you prefer)? If is too much hassle, no problem.

Thanks again for the support, ill update you soon on my results (I hope with positive results).

[gcarq]

I've indexed the blockchain from an 5400rpm HDD (with ext4) which yields about 130MB/s where the CPU had 25% iowait, so it could be even faster with a SSD.

It took about 2 hours to index the first 364K blocks :/ (Don't know if the slowness is hardware related or related to the probably corrupted .blk files ).

Are you running the bitcoin client in the background? Maybe the client locks and/or modifies those files at runtime, which would explain the I/O errors. Also I've never checked the parser with pruned blocks, so this could be an issue too. Nevertheless to exclude fs corruption make a quick check with fsck.

Unfortunately I don't have an up to date csvdump anymore (I accidently deleted it, lol), the next time I create one I can share it via torrent.

You're welcome, I'm curious if it works.

[Nellous]

Hello again,

Just wanted to say that after downloading the blockchain all together from one machine without stopping the download the extraction went smoothly. It took about 311 minutes for the whole 112 gb blockchain with no errors.

Thanks again for your help, it was really appreciated!

[gcarq]

I'm glad to hear that it works now, I'm closing this issue. Feel free to reopen if something doesn't work

cheers, Michael