Hard replay performance (v1.3.0)

[jacktang]

Hello,

I want to sync with mainnet and using offline archived data with hard-replay mode(v1.3.0). Now there are 20303573 blocks and the first 10000000 blocks are processed fast (2,000,000 blocks per hour), and the later become slow down(450,000 blocks per hour). And consume more RAM (75% of 8G). How can I improve the performance? Add more RAM? Thanks!

[taokayan]

8G is unlikely able to replay the whole block chain. What's your hardware config?

[jacktang]

@taokayan the test box is cloud os, here the hardware info

CPU: 2-core with Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
RAM: 8G
DISK: 300G

[taokayan]

I highly guess to have at least 32GB RAM and a faster CPU (3+GHz) to synced up with mainnet. Also disabling history_plugin and mongodb_pluign would make things faster.

[jacktang]

I upgrade CPU to 3.2 GHz and still the same issue. The ram keeps in 75%(8G) and CPU is only 13% busy, I check the I/O, the performance also is down, the peak of wkB/s could be 4000+

$ sudo iostat -x -p
Linux 4.4.0-117-generic (nextoken)  10/09/2018  _x86_64_    (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          26.70    0.00    1.58   20.83    0.00   50.89

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.06    1.47    0.23    32.27     1.43    39.54     0.01    3.36    3.24    4.13   0.52   0.09
vda1              0.00     0.06    1.47    0.23    32.22     1.43    39.57     0.01    3.37    3.24    4.17   0.52   0.09
vdb               0.00    27.59  836.07  777.42 19483.36  8192.06    34.31    11.64    7.22    1.35   13.53   0.26  42.04
vdb1              0.00    27.59  836.07  777.42 19483.33  8192.06    34.31    11.64    7.22    1.35   13.53   0.26  42.04

And some eos config

chain-state-db-size-mb = 48000
reversible-blocks-db-size-mb = 340

In order to watch some contracts transaction, I enabled history and history api plugins. Thoughts?

[jacktang]

I upgrade to 16G RAM, the last 800,00,000 blocks take 5-6 days again.

[jgiszczak]

Replay benefits from an instance capable of very high IOPS, especially when the history_plugin is enabled.

[jgiszczak]

nodeos now supports restoring from a snapshot and block log downloaded via out-of-band means. Install v1.4.2 and download snapshot and block log data from your desired EOS.IO-based network.

Note this technique only brings the nodeos instance up to date. If a database plugin is desired, replay is still required. This technique can still be useful in that case by rapidly bringing a local node into sync with the desired network, then using it as the source for local replay on a second node. Local network conditions may be directly controlled for optimal replay on the second database-enabled node.