search

bitcoinjs / indexd

An external bitcoind index management service module
ISC License
53 stars 23 forks source link

Isolate indexes (or at least 2nd order data calls) #26

Closed dcousens closed 6 years ago

dcousens commented 6 years ago

From https://github.com/bitcoinjs/indexd/issues/11

We should instead maintain two indexd chains, one for 1st order data, and the other chain, for the 2nd order data...

This would allow us to independently roll-back the chains in the event of failure.

This would also mean we can run a direct import using fast-dat-parser and allow the fee-data to catch up.

dcousens commented 6 years ago

This would mean we have two independent resync-like operations required for a typical indexd node.

dcousens commented 6 years ago

It would additionally mean a native offline/non-rpc fast import can get you 99% of the functionality required within minutes, and any 2nd order functionality (fee data! for now) would be sychronized as soon as possible.

KanoczTomas commented 6 years ago

this sounds really cool, i am indexing for a week now on an older node, and still 80k blocks to index ... this would be very welcomed, as we could use the *.dat files present on the node already!

dcousens commented 6 years ago

Perhaps it will be generic/easiest/simplest to maintain a tip for each index.

dcousens commented 6 years ago

In that way, we can add new indexes and have them "catch up" without having to re-sync every index in the database.

dcousens commented 6 years ago

Started work on this in https://github.com/bitcoinjs/indexd/tree/isoindex

To prevent performance loss, the db atomic can be shared between connect calls across indexes! :smiley:

Adding support for https://github.com/bitcoinjs/indexd/issues/27 is easy enough.

I think in terms of indexes updating themselves independently, it will be best to determine which chains have matched tips, then update them together.

Then, for 2nd order indexes that are really slow like the fee index, we could have a configuration parameter that allows you to continuously isolate it so that it doesn't slow down the other indexes (as much).

Yes, that may mean that it could take many hours more to index [for the fee index,] but the result is that the indexes that do not need to self-reference the database can insert insanely fast, 100's/1000's of blocks at a time.

Overall, I think it will be highly beneficial.

dcousens commented 6 years ago

Closing for PR, https://github.com/bitcoinjs/indexd/pull/37