search

blockchain-unica / blockapi

A general framework for blockchain analytics
http://blockchain.unica.it/projects/blockchain-analytics/
GNU General Public License v3.0
129 stars 64 forks source link

Missing ETH blocks from 2637146 to approx. 4696000 #9

Closed fdrobnic closed 11 months ago

fdrobnic commented 6 years ago

I'm testing import of the whole ETH chain into MongoDB. Blocks get imported except from height 2637146 onwards. Seems that the call web3j.ethGetBlockByNumber(new DefaultBlockParameterNumber(height), true).sendAsync().get().getBlock returns null for them. Successful reading resumes a little before height 4696000. Consequently, there are now almost 30 million documents (transactions) in my MongoDB but many may be missing.

fdrobnic commented 6 years ago

Update: if I try to import the block that didn't succeed before on its own then it works. Also works with setStart and setEnd setting which gives about one million blocks. I suspect the size of val blockchain could be the cause. I'm working on a laptop with 16GB RAM and everything locally.

fdrobnic commented 6 years ago

There may be problems with the Parity node. I tried to run the program with Parity node set to not sync the chain in order to not consume too much resources. After a short time when reading was successful the node stopped to return data and got into a loop so that I had to stop it manually. Then I started it up and reading was possible again.

fdrobnic commented 6 years ago

Installed the new version of Parity (1.7.12) and so far it looks promising. Import hasn't finished yet.

stefanolande commented 6 years ago

Hello @fdrobnic, any news with the new version of Parity?

fdrobnic commented 6 years ago

Well, it has gone further but stopped again at block 3822752. Now, even restart doesn't help. I need to investigate if there is something wrong with my Parity database.

fdrobnic commented 6 years ago

Using node and web3 library I can access all the blocks that appear problematic, one at a time. Also yesterday the reading went further and stopped. I repeated reading from that point on and it succeeded a few times and each time read a few thousand blocks and stoppped again. This way it reached to the blocks a liitle over 3900000. From that point on I couldn't get further and restart didn't help. Seems to me that Parity has to do its synchronization work for some time and do some housekeeping along the way and then the reading can go on. Just speculating.