Occasional block skips

[rmerom]

Thanks for writing this code, we're really happy with it.

Sometimes the callback our code has on etheruem-blocks is called for block X, then for block X + 2, never calling with X + 1, even in the future.

Do you have any idea why this might be happening?

[hiddentao]

Are you using it with a local private net with extremely low mining difficulty? I've had that happen to me in such cases, especially early on just after Genesis.

Director HiddenTao Ltd

www.hiddentao.com

(Sent from my mobile, apologies for brevity)

On 16 March 2017 17:43:49 Ron Merom notifications@github.com wrote:

Thanks for writing this code, we're really happy with it.

Sometimes the callback our code has on etheruem-blocks is called for block X, then for block X + 2, never calling with X + 1, even in the future.

Do you have any idea why this might be happening?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/hiddentao/ethereum-blocks/issues/1

[rmerom]

Thanks. We're working with Ethereum mainnet actually...

[hiddentao]

Well that's definitely not good then. Have you tried it with the logger on? This will give more debugging info.

https://github.com/hiddentao/ethereum-blocks#logger

[hiddentao]

Ultimately this relies on web3.js to pass it the new blocks - if a block is missing then web3 has failed to pass it through. I'm fairly confident of this because of the architecture I've put in..

Incoming blocks are places into a queue (an Array), and then a processing loop runs asynchronously at a set interval to go through all blocks in the queue. Since JS is single-threaded it should never happen that the queue gets modified in parallel from two different areas. The only line which removes blocks from the queue does so in a batch.

Having said that, my architecture may have a bug I'm as yet unaware of, and so I'm happy to be proven wrong.

[rmerom]

Thanks! I only somehow saw your answer now. I'll turn the logger on and let you know of any findings.

[rmerom]

Here's an example with the logger on (note some of the logging is by other parts of the code)

I 2017-04-01T04:19:47.019Z Got 1 block(s) to process 
I 2017-04-01T04:19:47.019Z Fetching block 0x877e0bd6bc8adf2843497872d14f3efe88671f720b7667b98f6f352d1ad862cf 
I 2017-04-01T04:19:47.021Z Processing block #3455553: 0x877e0bd6bc8adf2843497872d14f3efe88671f720b7667b98f6f352d1ad862cf ... 
I 2017-04-01T04:19:47.021Z Going to invoke 1 handlers for block event on block 0x877e0bd6bc8adf2843497872d14f3efe88671f720b7667b98f6f352d1ad862cf... 
I 2017-04-01T04:19:47.021Z Processed block: 3455553 
I 2017-04-01T04:19:47.021Z Invoked handler 'blockProcessor' 
I 2017-04-01T04:19:47.021Z Finished invoking handlers for block event on block 0x877e0bd6bc8adf2843497872d14f3efe88671f720b7667b98f6f352d1ad862cf 
I 2017-04-01T04:19:47.021Z ... done processing block #3455553: 0x877e0bd6bc8adf2843497872d14f3efe88671f720b7667b98f6f352d1ad862cf 
I 2017-04-01T04:20:08.981Z Got filter callback 
I 2017-04-01T04:20:08.981Z Got filter callback 
I 2017-04-01T04:20:09.982Z Got filter callback 
I 2017-04-01T04:20:10.983Z Got filter callback 
I 2017-04-01T04:20:10.983Z Got filter callback 
I 2017-04-01T04:20:12.024Z Got 5 block(s) to process 
I 2017-04-01T04:20:12.024Z Fetching block 0xd5e12ad5416e07eaf7faa47d040952f4b6479e7a6a3ac96652c4d270952d4a84 
I 2017-04-01T04:20:12.025Z Processing block #3455558: 0xd5e12ad5416e07eaf7faa47d040952f4b6479e7a6a3ac96652c4d270952d4a84 ... 
I 2017-04-01T04:20:12.025Z Going to invoke 1 handlers for block event on block 0xd5e12ad5416e07eaf7faa47d040952f4b6479e7a6a3ac96652c4d270952d4a84... 
I 2017-04-01T04:20:12.025Z Processed block: 3455558 

Please let me know if you need any other info.

[hiddentao]

From the logs I don't see what is missing. Did it receive and process the blocks in the right order? Perhaps additional logging is needed to find out in what order blocks are arriving.