nodeos --hard-replay appears to leak memory 1.6.0-rc1

matthewdarwin commented 5 years ago

When replaying blocks on mainnet with "/opt/eosio/bin/nodeos --hard-replay --replay-blockchain", nodeos seems to leak memory.

When I replay blocks from scratch, I run out of RAM (machine starts swapping) on a RAM=16GB CPU=i9-9900k machine in about 30 hours. If you restart nodoes then memory goes back to normal. If you restart nodeos regularly (every hour or so), replay will finish without running out of RAM.

I'm testing 1.6.0-rc1 in a docker container (using the equivalent to the out of box builder image, but I built my own). I did not test earlier versions.

Here is a chunk of log file from around block 15M. In 5 minutes, resident memory (RSS) went up by ~20K, while state file on disk (DISK) went up ~8K.

RSS: 1604 SIZE: 10993 DISK: 1285 DT: 2018-12-27 14:40:20
RSS: 1607 SIZE: 10993 DISK: 1286 DT: 2018-12-27 14:40:40
RSS: 1607 SIZE: 10993 DISK: 1286 DT: 2018-12-27 14:41:00
RSS: 1608 SIZE: 10993 DISK: 1286 DT: 2018-12-27 14:41:20
RSS: 1609 SIZE: 10993 DISK: 1286 DT: 2018-12-27 14:41:40
RSS: 1610 SIZE: 10993 DISK: 1287 DT: 2018-12-27 14:42:00
RSS: 1615 SIZE: 10993 DISK: 1288 DT: 2018-12-27 14:42:20
RSS: 1616 SIZE: 10993 DISK: 1288 DT: 2018-12-27 14:42:40
RSS: 1616 SIZE: 10993 DISK: 1288 DT: 2018-12-27 14:43:00
RSS: 1617 SIZE: 10993 DISK: 1289 DT: 2018-12-27 14:43:20
RSS: 1618 SIZE: 10993 DISK: 1289 DT: 2018-12-27 14:43:40
RSS: 1620 SIZE: 10993 DISK: 1290 DT: 2018-12-27 14:44:00
RSS: 1621 SIZE: 10993 DISK: 1291 DT: 2018-12-27 14:44:20
RSS: 1622 SIZE: 10993 DISK: 1292 DT: 2018-12-27 14:44:40
RSS: 1623 SIZE: 10993 DISK: 1293 DT: 2018-12-27 14:45:00

Here is 5 minutes from the replay where the on disk size of the state file says the same and the resident memory keeps increasing

RSS: 2003 SIZE: 11012 DISK: 1666 DT: 2018-12-27 16:55:00
RSS: 2006 SIZE: 11015 DISK: 1666 DT: 2018-12-27 16:55:20
RSS: 2007 SIZE: 11016 DISK: 1666 DT: 2018-12-27 16:55:40
RSS: 2007 SIZE: 11016 DISK: 1666 DT: 2018-12-27 16:56:00
RSS: 2008 SIZE: 11016 DISK: 1666 DT: 2018-12-27 16:56:20
RSS: 2009 SIZE: 11018 DISK: 1666 DT: 2018-12-27 16:56:40
RSS: 2017 SIZE: 11026 DISK: 1666 DT: 2018-12-27 16:57:00
RSS: 2020 SIZE: 11028 DISK: 1666 DT: 2018-12-27 16:57:20
RSS: 2024 SIZE: 11033 DISK: 1666 DT: 2018-12-27 16:57:40
RSS: 2024 SIZE: 11033 DISK: 1666 DT: 2018-12-27 16:58:00
RSS: 2025 SIZE: 11035 DISK: 1666 DT: 2018-12-27 16:58:20
RSS: 2026 SIZE: 11036 DISK: 1666 DT: 2018-12-27 16:58:40
RSS: 2028 SIZE: 11038 DISK: 1666 DT: 2018-12-27 16:59:00
RSS: 2031 SIZE: 11040 DISK: 1666 DT: 2018-12-27 16:59:20
RSS: 2033 SIZE: 11043 DISK: 1666 DT: 2018-12-27 16:59:40
RSS: 2039 SIZE: 11049 DISK: 1666 DT: 2018-12-27 17:00:00

here is the config.ini:

chain-threads = 2
wasm-runtime = wabt
chain-state-db-size-mb = 8192
reversible-blocks-db-size-mb = 2048
contracts-console = false
https-client-validate-peers = 1
http-server-address = 0.0.0.0:8888
http-validate-host = false
access-control-allow-credentials = false
max-body-size = 1048576
p2p-listen-endpoint = 0.0.0.0:9876
p2p-max-nodes-per-host = 1
agent-name = "EOS Nation"
allowed-connection = any
max-clients = 350
connection-cleanup-period = 30
network-version-match = 0
sync-fetch-span = 500
enable-stale-production = false
pause-on-startup = false
max-transaction-time = 30
max-irreversible-block-age = -1
keosd-provider-timeout = 5
txn-reference-block-lag = 0
abi-serializer-max-time-ms = 2000
verbose-http-errors = true
plugin = eosio::http_plugin
plugin = eosio::chain_api_plugin

matthewdarwin commented 5 years ago

This issue was discussed earlier today in https://t.me/eosfullnodes

Shrey33 commented 5 years ago

@matthewdarwin can I have the reproduction steps ?

matthewdarwin commented 5 years ago

Reproduction steps 1) get a copy of blocks 2) run nodeos --hard-replay --replay-blockchain" 3) wait

spoonincode commented 5 years ago

Expected mostly fixed with #7179

EOSIO / eos

nodeos --hard-replay appears to leak memory 1.6.0-rc1 #6533