seems btcd is way slower than bitcoin core in syncing?

[lzl124631x]

I remember I used bitcoin core to sync btc full node on EC2 (200+G) in one day. But I used btcd to sync full node -- two days later it just synced 108G ?

The EC2 config is the same.

[totaloutput]

@lzl124631x there are couple reasons that btcd is slower than bitcoind. The main reason I see is that btcd download from one peer only in bootstrapping the blocks, while bitcoind download from 8. So you need to find a closer and faster node to your ec2 instance. Let me know if you need more details, I can share what I did to speed up my syncing of 200+ G.. But it will never be as fast as bitcoind.

[jcvernaleo]

dcrd has an issue open related to multipeer downloads https://github.com/decred/dcrd/issues/1145 which appears to be partially done.

I haven't been following it closely enough to know if it is likely to be backportable to btcd but might be interesting to take a look at.

[totaloutput]

@lzl124631x : If your node is in US (like mine), your connection will be slow to the default nodes in btcd DNSSeed under chaincfg/params.go . If you just manually add couple nodes in US, the bootstrapping speed will be much faster. For example add this to your startup btcd command:--connection=157.131.198.183:8333 --connection==209.108.206.229:8333 . The IPs keep changing, you can use the leaderboard to find latest IPs close to your country.

I put more details in this post, if you want to look.

[lzl124631x]

@totaloutput Good to know that! But is bitcoind smart enough to sync with nodes geographical closer? If yes, why doesn't btcd have that yet?

[totaloutput]

@lzl124631x I don't think current bitcoind code is smart enough to find geographically closer nodes to sync. (I could be wrong) The current code is to find 1 node to get the headers of all the blocks, and find 8 nodes to ask for the content (data) of the blocks, if any of the node is slow/unresponse for 2 seconds, it will drop that node and ask for a new one. btcd code doesn't have that build in yet. It's a good feature, especially when the blockchain data is getting bigger and bigger. But majority of the btcd development resource moved onto dcrd (decred). My guess is that feature will be first implemented there then backport to btcd if someone wants to do it.

[l0k18]

I am using the btcd codebase to update the parallelcoin network (it almost even still has malleability vulnerabilities) and looked at the p2p syncmanager stuff.

I made a small alteration that favours selecting the lowest ping out of the connected peers, the code is here: https://github.com/btcsuite/btcd/blob/master/netsync/manager.go#L261 where you need to make the change and here: https://github.com/btcsuite/btcd/blob/master/netsync/manager.go#L261 - I personally would think that more metadata stored about peers could get a more accurate picture of available block sources, such as a record of the last block/second rate when the peer was synced from. Still, lowest ping of up to 8 currently connected nodes probably still helps.

btcd as delivered in the git repo has a series of checkpoints. When checkpoints are set, it pre-fetches the block headers and compact filters (if available). This likely actually slows things down during initial sync. I'd suggest trying to disable the checkpoints, to eliminate this double-downloading. Checkpoints are mainly to stop miners filling mempools with new blocks forked from ancient blocks, which are extremely unlikely to ever exceed the cumulative work total of the current best block. With my work on a forked chain, I am finding that when I set checkpoints, everything after gets orphaned. I am looking into this.

Since checkpoints main purpose, in effect, is simply to reduce time-wasting old forks from clogging up the mempools, it's my opinion that the more effective strategy would be to simply disallow forks that would require an inordinate amount of hashpower to push ahead of the tip. Someone mentioned this a long time ago in a btctalk post, the idea of simply disallowing new blocks to attach before some reasonable number of blocks previously. I think it should be on the basis of cumulative weight. A reasonable figure could be derived from the known hashpower for an algorithm as currently exists on all networks.

And yes, the btcd netsync library only downloads from one source at a time. I think it would make sense to improve this when it is syncing pre-checkpoint to establish several links and download several segments of the chain from several sources at once, while grabbing the blocks-only from a few nodes at the same time. The consensus chain could then be established earlier in the sync process and of course nonconforming blocks, the node serving them up should get a huge banscore increment for this, i mean, weeks of no contact, trying to propagate a fake chain is pretty serious.

[awb99]

This is a huge bug. I am already waiting a week for BTCD to get all the blockchain. And this should really only take a day maximum. BitcoinD is for sure 20x faster. How can one roll out any service if the bitcoin daemon is so slow to get started?

[ccconnor]

@lzl124631x : If your node is in US (like mine), your connection will be slow to the default nodes in btcd DNSSeed under chaincfg/params.go . If you just manually add couple nodes in US, the bootstrapping speed will be much faster. For example add this to your startup btcd command:--connection=157.131.198.183:8333 --connection==209.108.206.229:8333 . The IPs keep changing, you can use the leaderboard to find latest IPs close to your country.

I put more details in this post, if you want to look.

I set connect=127.0.0.1, but it took three days to sync to 333365, and it's getting slower and slower.

[justinmoon]

I'm getting about 1 block every 2 seconds. I tried modifying same addpeer and connect config parameters a few times but didn't see any change.

It was much faster than this for the first 100,00 or 200,000 blocks. Slowed down in the mid-high 200.000's ...

[ccconnor]

@justinmoon Same to me. At first, 200 or 300 blocks every 10 seconds; after 178411, less than 100 blocks every 10 seconds; now, somtimes 1 block every 10 seconds. Kick-ass!

[OlaStenberg]

@justinmoon Same to me. At first, 200 or 300 blocks every 10 seconds; after 178411, less than 100 blocks every 10 seconds; now, somtimes 1 block every 10 seconds. Kick-ass!

What version are you running? I pulled roasbeef/btcd, installed latest, but ended up with 0.9.0-beta, which had a lot of stalled sync problems, was processing 1 block every 2 minutes at mid 300k.

Trying 0.12.0-beta now, I was doing 1k per 10s, but just passed 180k and I'm barely doing 200 per 10s now.

[ccconnor]

@OlaStenberg I'm running master(version 0.12.0-beta).

[OlaStenberg]

@OlaStenberg I'm running master(version 0.12.0-beta).

Ok, seem to be working a lot better for me with master, even if it slowed down to 20 blocks every 10s (around 350k). At what height did it slow down to 1 block every 10s for you?

[ccconnor]

@OlaStenberg I started syncing from the begining again. At first, 17k blocks per 10s, now(204518) about 40 per 10s. And I find it's "fetchUtxosMain" that's so slow. Maybe there are some design flaws.

[begetan]

Btcd sync time is around 10x slower than bitcoin core 0.18 for now.

I did short profiling of the code in sync state and found that the incredible amount of CPU time is wasting by the garbage collector.

This is in turn the result of the LevelDB calls. In the same time Btcd is using very low amount of the memory. In nowadays even a small devices like Raspberry PI has enough amount of the memory.

I thing the right way of optimization is to improve the algorithm of LevelDB handling and to optimize malloc usage

[ghost]

For me it was dealbraker and I switched to bitcoin core :( I synced whole blockchain in little more than day. With btcd after few days I was still in the 60% of the height and like 30% of the size. :( It makes me sad, I would like to see more diversity but with this, I have no other option :(

[l0k18]

Two things stand out relating to this from my close work with the code on an old bitcoin fork (parallelcoin). One is the EC library. First thing I see is that the precompute could optionally be made a lot bigger, to improve performance, and on the other hand, also, using the fastest possible C library for koblitz curves optionally.

Secondly I don't think that all of the problem with the database is the leveldb back end, though there is probably issues there (again maybe cgo option?) and in between that and implementing the interface, common code exists also. This issue is common for Go. I watched a talk a few weeks ago about a video streaming service where they found they basically had to single thread the scheduling of the streams to reduce blocking, which of course points straight at the GC being incorrect for the task. It shouldn't be mysterious because the scheduler's main workload overall is IO bound and that's a bad fit for CPU bound stuff like transaction validation and database cache management.

Solving the problem probably will follow a pattern similar to the solution used by the streaming service, to explicitly schedule processing instead of trusting the scheduler, and minimising post init memory allocation with one big early allocation. All of which are a lot of work. The GC percent is default set to 10% and this is quite generous but still puts a ceiling on throughput, which conflicts with latency, in almost all cases.

[Rjected]

In my profiling, garbage collection and runtime operations were taking up a lot of CPU time during sync, which was worrying - so I profiled some more (for allocations), and immutable treap operations were by far the biggest allocators, so that may be the issue.

Here's what I have for allocations when syncing 2012 blocks:

File: btcd
Type: alloc_space
Time: Aug 16, 2019 at 2:53pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 182.87GB, 85.76% of 213.25GB total
Dropped 379 nodes (cum <= 1.07GB)
Showing top 10 nodes out of 109
      flat  flat%   sum%        cum   cum%
   83.99GB 39.39% 39.39%    83.99GB 39.39%  github.com/btcsuite/btcd/database/internal/treap.cloneTreapNode
   64.99GB 30.48% 69.86%       65GB 30.48%  github.com/btcsuite/btcd/txscript.parseScriptTemplate
    6.75GB  3.16% 73.03%    34.08GB 15.98%  github.com/btcsuite/btcd/database/internal/treap.(*Immutable).Delete
    6.34GB  2.97% 76.00%     6.34GB  2.97%  github.com/btcsuite/goleveldb/leveldb/table.(*Reader).newBlockIter
    5.06GB  2.37% 78.37%    10.78GB  5.05%  github.com/btcsuite/goleveldb/leveldb/util.Hash
    4.92GB  2.31% 80.68%     4.92GB  2.31%  bytes.NewBuffer
    3.25GB  1.52% 82.20%     3.25GB  1.52%  github.com/btcsuite/btcd/database/internal/treap.newTreapNode
    2.87GB  1.35% 83.55%     2.87GB  1.35%  github.com/btcsuite/btcd/wire.(*MsgTx).BtcDecode
    2.52GB  1.18% 84.73%    16.38GB  7.68%  github.com/btcsuite/btcd/blockchain.(*UtxoViewpoint).addTxOut
    2.20GB  1.03% 85.76%     2.20GB  1.03%  github.com/btcsuite/btcd/database/internal/treap.(*parentStack).Push

For cloneTreapNode:

(pprof) list cloneTreapNode
Total: 213.25GB
ROUTINE ======================== github.com/btcsuite/btcd/database/internal/treap.cloneTreapNode in /home/dan-server/btcd/database/internal/treap/immutable.go
   83.99GB    83.99GB (flat, cum) 39.39% of Total
         .          .     14:   return &treapNode{
         .          .     15:       key:      node.key,
         .          .     16:       value:    node.value,
         .          .     17:       priority: node.priority,
         .          .     18:       left:     node.left,
   83.99GB    83.99GB     19:       right:    node.right,
         .          .     20:   }
         .          .     21:}
         .          .     22:
         .          .     23:// Immutable represents a treap data structure which is used to hold ordered
         .          .     24:// key/value pairs using a combination of binary search tree and heap semantics.

And parseScriptTemplate:

(pprof) list parseScriptTemplate
Total: 213.25GB
ROUTINE ======================== github.com/btcsuite/btcd/txscript.parseScriptTemplate in /home/dan-server/btcd/txscript/script.go
   64.99GB       65GB (flat, cum) 30.48% of Total
         .          .    193:
         .          .    194:// parseScriptTemplate is the same as parseScript but allows the passing of the
         .          .    195:// template list for testing purposes.  When there are parse errors, it returns
         .          .    196:// the list of parsed opcodes up to the point of failure along with the error.
         .          .    197:func parseScriptTemplate(script []byte, opcodes *[256]opcode) ([]parsedOpcode, error) {
   64.99GB    64.99GB    198:   retScript := make([]parsedOpcode, 0, len(script))
         .          .    199:   for i := 0; i < len(script); {
         .          .    200:       instr := script[i]
         .          .    201:       op := &opcodes[instr]
         .          .    202:       pop := parsedOpcode{opcode: op}

etc. etc. nothing else significant

So these combined account for about 70% of allocations, but account for less than 10% of in-use space at runtime. In both cases, keeping some state so we reduce the number of allocations (and deallocations) would be beneficial. I bet replacing the immutable treaps in the dbcache would really help sync speed. LevelDB calls are also fine, they are all fairly lightweight - IMO the issue is the cache. More profiling probably needs to be done, but my guess is the dbcache.

[matt24smith]

I'm trying to set up my own node with btcd instead of bitcoin core, since I want to support a healthy blockchain ecosystem. But after two weeks trying to sync the mainnet blockchain, it's just way too slow. I can't recommend anyone use this software when it takes so long just to get started. Sync times are increasingly slow as the block height rises - see the attached image.

(The gap on nov. 11 - 13 was from when I stopped syncing to mess with configs)

[l0k18]

I will be focusing on solving this problem in my project (github.com/p9c/pod), but thanks to Rjected I also have looked at some of the treap code as well as the script engine and I am pretty sure both have got serious garbage accumulation problems. Optimising them and aiming for zero runtime allocation will probably go a long way towards a solution.

Incidentally, a part of btcd that I have had INTENSE work with is the CPU miner. It uses two mutexes, and stops and creates new goroutines constantly. I built a library that allows me to attach an RPC to the standard output, built a small, dedicated single thread (two threads but primary work), I use two channels, stop and start, and I thought I would need an atomic or compareandswap but it turns out just using two channels and a for loop with a 'work mode' second loop, both parts of the loop drain the channels not relevant (the runner ignores the run signal channel and the pauser ignores the pause signal channel), and the lock contention is obviously so bad that a little over 10% of potential performance is chewed up with synchronisation.

I know well enough from what I saw of the script engine and database drivers/indexers that the programmers who write it are obviously former C++/Java programmers because they pretty much mainly rely on mutexes for synchronisation and mutexes, which are the slowest sync primitives, and where channels are used, in places that I would expect to see async calls used in these older, less network-focused languages.

For bitcoin forks, especially small, neglected ones like parallelcoin, its sync rate is fine. 8 minutes on my Ryzen 5 1600/SSD/32gb machine, at a height of about 210,000. The chain barely has maybe 2 transactions per block, on average. But even still, at 99000 and again around 160000 it bogs down badly and appears to be mainly garbage collecting, so with the typical block payload of bitcoin, I imagine the complexity of the chain of provenance of tokens explodes exponentially, and that graph exactly shows this pattern.

I'm not sure where I will start with it, but I strongly suspect write amplification is also hiding in there, a performance problem well known to be existing with LevelDB, and even RocksDB and BoltDB, and resolved in Badger, so first step will be building a badger driver. I'd guess that especially as the number of transactions grows that write amplification is causing an issue every time the database updates the values it has to write the keys again as well, combined with the geometric rise in complexity of validations to confirm they correctly chain back to the coinbase.

Second thing I expect to look at is the treaps. There is some parts of btcd that attempt to eliminate runtime allocations, at least one buffer freelist, but there is a lot of creation of small byte slices that are discarded later. As tends to be the case with Go, the naive (I did mention the mutexes, they are a naive use of Go) implementation does not take into account GC or thread scheduling, and when the bottlenecks are really bad, usually it means you have to take over both memory and scheduling work from the Go runtime to get a better result. I already saw one clear example of this just in the use of isolated processes connected via IPC pipes and using two channels and one ticker instead of 2 or 3 mutexes and 3 different tickers made it produce more than 10% more hashes.

If anyone is interested who is following, keep an eye on the repo I mentioned above as over the next 6 months I will be focusing on optimizing everything. I am nearly finished implementing the beta spec for my fork, and I have aimed to make it accessible enough and not stomping over top of too much of what is already there that differs for the chain I am working on. I am still a bit lost as to how to enable full segwit support, and you will see I have merged btcwallet, btcd, btcutil and btcjson repositories into one, created a unified configuration system, and mostly done and robust handling of concurrent running of wallet and node together for a more conventional integral node/wallet mode of operation. I understand some of the reasons behind so vigorously separating them but in my opinion in the absence of a really good SPV implementation makes doing this a step backwards.

Based on watching CPU utilization and a little bit of profiling I can see so much empty spaces between with CPU doing literally nothing for about 60-70% of the time, during the sync process, so I am very leaning towards the idea that synchronization is the bigger issue, and second is garbage generation, and thirdly, write amplification due to the database log structure, with updating metadata related to block nodes in the database.

[l0k18]

By the way, out of curiosity I tried disabling the addrindex and txindex while on initial sync. Not only did it not seem to take any less time, the amount of metadata generated in the database folder looked exactly the same as though it was on. I am not sure if it is ignoring my configuration change, I checked through all the places where that setting is read and it definitely should not have been running the indexers.

[btcbobby]

Looks like a memory ballast could be in order... https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/

[l0k18]

I remember reading this one before. I am going to try it out with my fork at https://github.com/p9c/pod and I will report back if it gets that kind of result (it is on a different, much smaller change but even still appears to have at least one big GC cleanup every 50-100,000 blocks and that is at least part of the problem for sure, as one or two blocks end up taking like a minute to process.

[jakesylvestre]

Hey @l0k18, what ended up happening? If it worked I'd be happy to port here since this has become a much bigger issue

[jakesylvestre]

So the ballast (10gb) does appear to substantially increase the garbage collection rate: Ballast: No ballast:

It looks like the greatest decrease here comes from: Ballast without debug.SetGCPercent(10): No Ballast without debug.SetGCPercent(10):

This appears to result in a 3x increase in heap size: 10% GC limit w/ ballast: 10% GC limit w/ no ballast: No Limit no ballast No Limit

[jakesylvestre]

I'll need to do a full sync to confirm, but it looks like a 3x increase in memory can get us a 3-4x increase in block sync speed. Ballast does not appear to improve allocation speed

[l0k18]

I have been working on a fork coin using this engine and I found it maxed out about 2000tx/s syncing from a node on my lan. I suspect this may be structural limit caused by signature verification performance. It has been a couple months since i was working on it so my memory might be hazy but i think it got best performance with gc limit at 100% which i think is 'no limit'.

I hadn't tried the ballast, and in a couple of days i can confirm.

[jakesylvestre]

Yeah, that's what I'm thinking. I think this is going to come down to a memory/cpu trade off. Maybe we can add a config option in here

[l0k18]

since only vps nodes may need to squeeze into under 1/2gb and nearly no home users have less than 4gb it is an option only needed for the former case but it is not difficult to defer the gc/ballast settings after flags/config are parsed.

For comparison, what is the bitcoin core memory usage anyhow? 2k tps is decently fast so long as network isn't the bottleneck. It also will indicate how many bytes the p2p side needs to be delivering to keep the cpu humming

[jakesylvestre]

Yeah, so I think the goal here is to make network the bottleneck. Tough to do, but that's probably the stretch goal.

Bitcoin Core recomends 2gb of ram. I've personally never seen it use that little, our current node is about 5gb

[l0k18]

2gb??? hah! btcd runs in under 400mb... see, that confirms what I thought - the base problem in btcd is memory contention, possibly caused by mutexes or the db driver needs to replicate the me ory cache so multiple threads can access relevant memory

[l0k18]

My forked version has a currently entirely unaltered tx validation engine and the thing that gets the 800tx/s average peak 2000tx/s (I think that is network limit) is this code:

// Main is the main entry point for pod
func Main() {
    runtime.GOMAXPROCS(runtime.NumCPU() * 3)
    debug.SetGCPercent(10)

I thought I should revisit the runtime parameters setting code to answer the question about how I got it to run faster, and I think that setting runtime.GOMAXPROCS to 3x was something I experimented with and eventually found it was the only thing that improved performance noticably. So the problem has to do with the fact that Go's scheduler was stinging out on the number of threads required to enable the maximum throughput for tx validation. ymmv of course but it's a simple change.

Most people don't realise that Go is concurrent not parallel scheduled. I also wrote a stand-alone miner which I created a pipe IPC to control multiple concurrent threads (each miner only gets one thread, btw, as it only needs one) and saw a 20% improvement just by shiifting scheduling from the runtime to the kernel. I think by raising the limit on goroutines you improve parallelism, which is where the C++ version in Bitcoin Core is probably performing better at this. I don't know how else this could be improved except by adding a feature to Go's goroutine scheduler to specify parallelism as the target instead of concurrency. Again, most people don't realise that all goroutines can multiplex equally well with one thread as with multiple. Selectors are randomised always (this is an omission from its implementation of CSP) in order to account for this, and why channels usually are unloaded (as opposed to loaded) in random order.

These are design decisions the Go authors made for specific reasons but they interfere with performance in limited cases, this is obviously one of them, allowing more threads to be spawned obviously helps ensure if there is a possibility to run more threads than there is CPU threads, that that code actually runs in parallel instead of being bottlenecked.

At least that is my best estimation of the situation.

[Colman]

Has there been any update on this problem? With the txindex and addrindex set to on, I'm syncing 1 block every minute! At this rate it will take over a month to sync. This makes the software completely unusable. If the problem isn't being worked on, is there any other stable libraries that offer the the txindex and addrindex options?

[jakesylvestre]

@Colman I've never seen it that slow. This sounds like it might be a disk i/o issue. Are you using some kind of NAS?

Alternatively - there are a few high impact PR's that would go a long way toward speeding this up. The most notable is probably #1373. It'll depend on your disk i /o but as @Roasbeef point's out that PR gets sync >24 hours. Would be great if you could help get that merged.

As far as other servers, it looks like you could check out this fork bitcoin/bitcoin#14053

[Colman]

Well I'm using Amazon's EBS which I think is considered NAS. I've tried their regular SSD and their high performance one which both yielded the same result. As far as I know, the latency between the machine and the EBS is <10ms, which IIRC, is the bottleneck right? Either way, why does the hard drive affect it that much? As far as I know, you're only accessing it to modify the UTXO set, TX index, and address index which can be done in batches.

Also, I checked out the @Roasbeef fork and even if a UTXO cache was implemented, I still don't get why it's so slow without one.

[jakesylvestre]

In my experience - it has a pretty big affect. We've got like 6 btcd nodes running in gcp and we had to switch them all over from pd-standard (7 days) to pd-ssd (4 days) and finally got down to 2 with an nvme persistent disk. I haven't benchmarked #1373 on these different disk types.

I don't remember the exact numbers but bitcoind has a similiar step function with the sync times being much lower (but differing commensurately) across the board

[Colman]

Okay I looked in to the specs for the AWS disk that I'm using and it's quite similar to yours. However, mine has been syncing for a month and still isn't done so obviously something else is up. What's your CPU and RAM specs? And are the timelines you gave me measured with the txindex and addrindex settings set to on? I'd assume they would increase the sync time.

[begetan]

@Colman If you are using a general-purpose EBS volume ( gp2) you can only get 1500 IOPS and 250 Mbps baseline speed for the volume of 500Gb, and lower for a smaller volume. This is far below any SSD disc, so the disk IO may be a root cause of slow work in this case.

You may try to use provisioned IOPS mode which is more expensive but closer to SSD performance.

[Colman]

Okay good to know thanks. I tend to avoid the IOPS ones because the last time I used it, I got charged like $120 just for the month which is totally outrageous. They probably pay off that drive in less than 2 months.

[Colman]

Also, is there a way to make RPC calls while the blockchain is syncing? I would like to see if the tx index will be fast enough to use before I spend all this money on the drives to sync it.

[jakesylvestre]

n1-cpu-4, but there is a fair bit of variance here in our llvm setup. I actually wrote a custom provisioner to combine the local disks into a single logical volume and iops increases with the number of disks

[asemoon]

This problem still exists in 2021! It is taking more than one day for me sync the blocks for 2017

[BrannonKing]

I want to confirm that my memory-pressure measurements today show that the cloneTreapNode is the chief culprit. The second worst is the ecdsa.Verify; there are a surprising number of make calls deep in big.nat. The third worst is the leveldb find function.

[pakar]

To anyone having performance issues when syncing you could try to set: (if you have enough memory that is) sigcachemaxsize=8000000

It seems to have dropped from 1 block per 60-90 seconds down to 10-20 seconds for 1-3 block's, with a few hiccups here and there..... Unless there was some huge drop in block-complexity around this height. height 524776, 2018-05-28 09:58:25 +0200 CEST

Summary of the behavior i can see on my system and some "gut-feeling" around a possible culprit that we might want to investigate, but too unfamiliar with the btcd codebase to do anything myself..

I'm doing a import on a 4 core 3.2Ghz with 32Gb RAM and 4x4TB HDD's running in raid10.

Funny thing is that i started a bitcoind sync, to the same filesystem, and it fully synced in 2 days while btcd is still at mid 2018 after over a week. ;)

One thing i have been thinking about is the lack of O_DIRECT fs access in go... Usually databases bypass the buffercache for writes (at least) to databases to reduce the memory fragmentation of pending writes.. Not sure if that's related to the issue here since my Dirty blocks usually stays around 10-15MB in /proc/meminfo) Currently the only application that is doing io against these disks btcd, with a few read-spikes of bitcoind from every 30-60 seconds or so.. IO usage by btcd, as reported by iotop, stays between 85-100% with ~500KB/s read and ~4-6MB/s writes and IOP's for the raid10 jumps around between 70-140 tps. in iostat. CPU usage of btcd jumps up and down all the time, but never breaks 100% so go routines does not seem to be used that much?

If wanted i can make a copy of the blockchain and run any tests you might want at this height... Just give me a shout...

[strusty]

@pakar I think it would be great if you uploaded a snapshot of the btcd chain for posterity, especially if this situation is not going to improve anytime soon.

[pakar]

@strusty Only been syncing during idle-time on the system so not yet fully synced. Currently at block-height 575734 (2019-05-12) and DB is at 301Gb and growing. I can upload to FTP / HTTP if someone wants to host it.

Create a new issue and reference me in it if i should do something.

[strusty]

@pakar I will set this up, and await the good word from you when you have it all synced. I will let you know here once it is ready.

[Rakeshnohria]

Hi @pakar ,

As discussion with @strusty , we are ready with the FTP details. Please let us know how we can share the details with you.

Thanks

[rtreffer]

Ok, same issue here, btcd is ridiculous slow.

This seems to be mostly goleveldb related (as others have stated). The following PR seems highly relevant: https://github.com/syndtr/goleveldb/pull/338

bitcoind configures leveldb based on a memory flag. I am using WriteBuffer: defaultCacheSize, BlockCacheCapacity: 2*defaultCacheSize for similar results. It would be nice to have a flag for cache / memory usage to fine-tune this.

I am also using NoSync: true for the initial sync. And syncing from bitcoind on localhost.

That said the initial sync is still days off but at least it is making decent progress.

[rkfg]

Another idea: I ran btcd with GOGC=500 and my block processing speed went up from 60-70 to 200+ (in 10 seconds) while downloading at around height 290000. This parameter makes GC trigger when the new heap is 5x bigger than the heap size since last collection. The default is 100 (in percents). This value can be increased further if you have enough memory. See the GC work by setting another env var GODEBUG=gctrace=1.

It's not a solution of course but a mere workaround. A real solution is to stop producing so much garbage during IBD but I understand it might be harder than it sounds.