Block processing taking forever

[satorsq]

Running blockstored on ubuntu 14.04. Using onename's default bitcoind server.

Blockstored creates its initial index and then proceeds to process blocks. Each block processing step takes 30seconds to 2minutes.

Here is the console output I receive after running for 10 minutes: jonathan@mark1:~$ blockstored start Starting blockstored server ... Do you have your own bitcoind server? (yes/no): no Using default bitcoind server at btcd.onename.com Creating initial index ... Processing block 343883 block_nameops [] Processing block 343884 block_nameops [] Processing block 343885 block_nameops [] Processing block 343886 block_nameops [] Processing block 343887 block_nameops [] Processing block 343888

Any thoughts on what might be causing this?

Thanks,

Jonathan

[muneeb-ali]

There are actually multiple things going on here, each one of them can/should be separated into a Github issue:

1) The Onename bitcoind server is running behind a load-balancer which makes it more reliable but also slower than if we expose the bitcoind without the load-balancer. Amazon's load-balancer needs to be "warmed up" by hitting it with traffic before it really starts to give better performance. There must be a workaround to this (can always have a custom load-balancer that will not have this issue) and need to figure out a solution to this initial slow-start time.

2) We're fetching blocks from the load-balancer sequentially right now. It's a no-brainer to either make this async or use threads. @jcnelson had some good ideas about this.

3) We're making calls to bitcoind for fetching each transactions meaning you first get the block (1 rpc call), and then if there are n transactions in the block, you make another n calls. There has to be a better way for doing this. @bmuller had some good ideas about this.

4) Even with all the above things are fixed, the reality is still that you need to get the blocks and then process them for name operations. At this stage, a lot of people just want to quickly start playing around with blockstore and see how it works. A docker that has pre-processed the blockchain upto a recent-enough block can be interesting. You'll have to trust the docker image and the maintainer, but that's fine if you treat it as just a way to quickly get up and running (you can also force the docker to re-analyze the blockchain and construct the namespace again before pushing it to production).

[jcnelson]

Hi @satorsq,

I just pushed branch 'parallel-startup' that should make the initial indexing considerably faster.

If you're not running bitcoind locally, feel free to point blockstored to our new load-balanced bitcoind fleet at 54.237.72.244 (which should be somewhat less variadic than btcd.onename.com). I regularly see one block per 1.5 seconds (on average) with this branch, as well as between 10Mbps to 20Mbps downstream RPC bandwidth.

-Jude

[jcnelson]

Since there has been no activity on this for some time, and since this has been long since fixed by having multiple processes index the blockchain in parallel and by installing a much more performant load-balancer, I'm going to close this issue. Please re-open if there are any further questions.