search

behas / bitcoingraph

A Python library for exploring the Bitcoin transaction graph.
Other
64 stars 21 forks source link

Requirements, time and neo4j format #7

Open v4lenz opened 8 years ago

v4lenz commented 8 years ago

I will soon try this tool but I would like to understand in advance three things:

  1. What are the requirements in terms of CPU, Disk and RAM, to run the whole thing?
  2. How much time would it get to fully sync the blockchain and export in Neo4j?
  3. In the Neo4j graph, what do nodes and edges represent? Addresses and transactions? Or it's a multi-graph with transactions as nodes and inputs/outputs as edges?
v4lenz commented 8 years ago

So, nothing?

k15z commented 7 years ago

Expect at least 256 gigabytes of disk space, give it a few days to process on a m1.medium instance.

zenonp commented 1 year ago

I just did blocks 0-677000, i.e. up to March 2021. The chain itself in $HOME/.bitcoind (up to current height) takes 530 GiB, the csv files for 0-677000 another 669 GiB, and then neo4j estimates it will need yet another ~590 GiB for the database.

Fully syncing the blockchain depends on when you do it (it keeps growing every ten minutes) and the speed of your internet connection. Producing the csv files depends on the speed of your disk(s), on your CPU and to a lesser extent on available RAM; on my AMD 5600G and a WD40 HDD it took about 40 hours. You can probably speed it up by keeping source and destination (blockchain->csv and csv->neo4j) on different disks and by not using copy on write on the target filesystem. Importing into neo4j will probably need 32 GB RAM, perhaps even more. When you start the import you get some suggestions for heap and other memory. Make very sure that you don't use swap.

v4lenz commented 1 year ago

Now I know. It took 7 years, but totally worth the wait 😉. Jokes aside, I remember having this need and back then requirements were of course a bit more forgiving. Not on this topics these days but it's nice to see there's still people investing their time in such tools.