search

straumat / blockchain2graph

Blockchain2graph extracts blockchain data (bitcoin) and insert them into a graph database (neo4j).
GNU General Public License v3.0
149 stars 32 forks source link

Performance after block 250k-300k #599

Open fbruzzaniti opened 4 years ago

fbruzzaniti commented 4 years ago

It appears that the more records that are inserted the slower the process get's, performance seemed to drop off sharply after block 250k-300k. Just looking at the GUI it appears to be the "saving block" step that takes the longest. Are there any tweaks or changes I can make ? The I'm using has 20 cores, 96 GB pf memory and has all the docker containers (straumat/blockchain2graph-bitcoin-neo4j:3.0) running locally.

straumat commented 4 years ago

Sadly no :( i did not find any way to make it faster... it's more of a neo4j issue I think :(

fbruzzaniti commented 4 years ago

Could it be the indexes slowing down writes (as they need to be updated) as mentioned here: https://stackoverflow.com/questions/18212860/neo4j-slow-savin] ? Are they needed for the import or can they be deleted and re-created after the import ?

straumat commented 4 years ago

Yes, indexes slow down the writes but to make analysis, before saving a new block, I link all transactions to their outcomes so I guess it will make things very slow to retrieve all the output transactions before saying. You can try to delete them and recreate them after if you want to try.