firehose-antelope really slow when replaying kylin and telos blocks

pinax-network / firehose-antelope

Firehose implementation for Antelope

Apache License 2.0

8 stars 1 forks source link

firehose-antelope really slow when replaying kylin and telos blocks #38

Open matthewdarwin opened 1 year ago

matthewdarwin commented 1 year ago

Just start from a snapshot from a few days ago. Watch the amount of CPU use by firehose and how slow the blockchain syncs forward (3 minutes for every 10 seconds in blocks). Without firehose there, the chain runs like 10x faster (30 minutes in 10 seconds). Any optimizations to be made?

Note: this is reader+merger running together.

matthewdarwin commented 1 year ago

Turning off merger doesn't make it faster.

fschoell commented 1 year ago

Is this limited to Kylin and Telos or does this impact all chains?

matthewdarwin commented 1 year ago

I didn't check the others, probably applies to all?

fschoell commented 1 year ago

Spend some time debugging this. It seems like:

The new code uses approx 2x the CPU time in comparison to the old code
The increased CPU time seems to be spend on garbage collection
The heap usage of the old code is about 2.5x the size of the new code

It seems like the old code is utilizing more heap in favor of less CPU time. I'm not sure how though.

It might also be a possibility to tweak the garbage collector here using GOGC. As it seems like:

doubling GOGC will double heap memory overheads and roughly halve GC CPU cost

fschoell commented 1 year ago

Some quick testing on the new code. CPU profiling for a 30s interval reports:

around 14s of CPU time used on GOGC=100 (default)
around 3.5s of CPU time used on GOGC=500
around 3s of CPU time used on GOGC=1000

in comparison it's around 7-8s of CPU time on the old code without GOGC tweaking.

So obviously need to do some testing when replaying blocks instead of getting this data in live mode, but we should definitely try this. Will increase memory usage, but this shouldn't be a big issue here I assume.