Rebase bitcore patches to 0.14

karelbilek commented 7 years ago

I rebased bitcore patches and made some corrections in the tests (the tx test was not running, because there is a built-in fee limit now) and in the code (changes in serialization calls, changes in segwit API, ...).

Since there was a big refactoring from 0.13.* to 0.14, I had to move the changes from main.cpp to other files; so I had to change the original bitcore patches slightly.

Both the qa tests and the c++ tests should be passing.

I don't know to which branch to put this PR, so I put it to master (which is an outdated version of upstream apparently), do as you please with it. (The commit list is giant, since it includes all the bitcoin commits between 0.12 and 0.14, plus the patches to bitcore.)

Since you have abandoned bitcore, I don't hope you will merge this, but maybe it will be helpful for others.

karelbilek commented 7 years ago

I am testing the address index on segwit testnet and it seems like it is not behaving correctly with segwit transactions. :/

edit: nevermind, error on my end. Segwit txs are running fine.

FrancisPouliot commented 7 years ago

THanks so much @runn1ng ! @gasteve some attention to this would greatly increase your users' confidence that Bitcore project has not been abandoned.

ghost commented 7 years ago

Personal blockchain explorers are important! Please keep this project alive!

FrancisPouliot commented 7 years ago

@gasteve seems like all is left to do here is merge? I think there have been further upgrades by @runn1ng since this pr

karelbilek commented 7 years ago

For some reason, while this worked fine on testnet and regtest, it got strangely slow on livenet, and the database is strangely sluggish even on SSH while downloading the blockchain for the first time; when the chain is about 50GB big, it syncs chain only 10 minutes from 1 hour and the rest is doing something, probably in leveldb.

However, it does work in the end, only there is this heavy performance hit.

I don't remember this happening in older version; so there might be some errors in the rebase; I will investigate later, but now don't have the time unfortunately.

(My personal, long-term plan is to get to know this code even more deeply and then try to get it into actual bitcoin core, but I am not sure I will have time for that either :D )

karelbilek commented 7 years ago

Some people are linking to this for some reason .... just FYI

The issue with blockchain download is a leveldb "issue". Leveldb is not designed to have that many indexes that big, so it struggles with ~100.000 files. It gets slow only on the initial synch, then it is fine; but the initial synch takes days. If you look at the code, it adds a separate key to the database for every output in the history, twice. It is 150 GB in 2 MB files.
The PR to bitcoin core was added here and basically dropped - https://github.com/bitcoin/bitcoin/pull/10370
The actual code is here - https://github.com/satoshilabs/bitcoin , https://github.com/satoshilabs/bitcore, and all the packages on satoshilabs organization that start with bitcore-* or insight
Also patched Bitcoin ABC (Bitcoin Cash) is here - https://github.com/satoshilabs/bitcoin-abc - and the "main" repo for that is just this subtree https://github.com/satoshilabs/bitcore/tree/bcash

I however have to say I cannot promise we will keep developing it long term or even add features etc; we are keeping it on "life support" so we can use it as a backend for Trezor Web Wallet.

karelbilek commented 4 years ago

oh the memories. Closing this though.

bitpay / bitcoin

Rebase bitcore patches to 0.14 #43