nodeos sync initializing chain plugin frozen in v1.2.4

[spartucus]

I'm using lastest version v1.2.4 to sync from mainnet, it looks like stopped at startup, the output log is just one line of "initializing chain plugin", no more logs, and nodeos process is alive when i'm using top to see its memory and cpu usage.

~/eos/build/programs/nodeos$ ./nodeos --config-dir mainnet_config/ --data-dir mainnet_data
2018-09-06T08:15:48.893 thread-0   chain_plugin.cpp:317          plugin_initialize    ] initializing chain plugin

And same config and command is fine with v1.2.3. So i think this may be a bug introduced in v1.2.4? Perhaps 3947e68106bea069df8c74631b7cc28eedebd165 ?

config.ini

    #producer-name = !!!!YOUR_BP_NAME!!!!!!!!
    #signature-provider = YOUR_BLOCK_SIGN_PUB_KEY=KEY:YOUR_BLOCK_SIGN_PRIV_KEY

    agent-name = AgentName

    #blocks-dir = "blocks"
    chain-state-db-size-mb = 30720
    chain-state-db-guard-size-mb = 50
    reversible-blocks-db-size-mb = 500
    contracts-console = false

    http-server-address = 0.0.0.0:80
    p2p-listen-endpoint = 0.0.0.0:9876
    #bnet-endpoint = !!YOUR_ENDPOINT_IP_ADDRESS!!:9875
    #bnet_threads = 4

    http-validate-host = false
    verbose-http-errors = true  
    abi-serializer-max-time-ms = 2000  

    access-control-allow-origin = *
    # access-control-allow-headers =
    # access-control-max-age =
    #access-control-allow-credentials = false

    #p2p-max-nodes-per-host = 4

    # actor-whitelist =
    # actor-blacklist =
    # contract-whitelist =
    # contract-blacklist =
    # filter-on =

    # action-blacklist = eosio::setcode

    # SSL
    # Filename with https private key in PEM format. Required for https (eosio::http_plugin)
    # https-server-address =
    # Filename with the certificate chain to present on https connections. PEM format. Required for https. (eosio::http_plugin)
    # https-certificate-chain-file =
    # Filename with https private key in PEM format. Required for https (eosio::http_plugin)
    # https-private-key-file =

    allowed-connection = any

    log-level-net-plugin = info
    max-clients = 150
    connection-cleanup-period = 30
    network-version-match = 1
    sync-fetch-span = 2000
    enable-stale-production = false

    max-implicit-request = 1500
    pause-on-startup = false
    max-transaction-time = 30
    max-irreversible-block-age = -1
    txn-reference-block-lag = 0
    unlock-timeout = 90000

    plugin = eosio::chain_api_plugin
    plugin = eosio::history_plugin
    plugin = eosio::history_api_plugin
    plugin = eosio::chain_plugin
    plugin = eosio::bnet_plugin
    plugin = eosio::net_api_plugin

    #plugin = eosio::mongo_db_plugin

    #bnet-connect =
    #bnet-connect =
    #bnet-connect =

# argentinaeos: AR, eosargentina 
p2p-peer-address = p2p.eosargentina.io:5222 

# aus1genereos: AU, Sydney 
p2p-peer-address = p2p.genereos.io:9876 

# bitfinexeos1: GB, London 
p2p-peer-address = eos-bp.bitfinex.com:9876 

# bitspacenode: GB, London 
p2p-peer-address = p2p.eos.bitspace.no:9876 

# blockmatrix1: US, N. Virginia 
p2p-peer-address = eosapi.blockmatrix.network:13546 

# cryptolions1: DE, Germany-Finland 
p2p-peer-address = bp.cryptolions.io:9876 

# cypherglasss: US, Minneapolis, MN 
p2p-peer-address = publicnode.cypherglass.com:9876 

# eos42freedom: GB, London 
p2p-peer-address = p2p.eos42.io:9876 

# eosbeijingbp: JP, Tokyo 
p2p-peer-address = bp.eosbeijing.one:8080 

# eoscanadacom: CA, Montreal, Canada 
p2p-peer-address = peering.mainnet.eoscanada.com:9876 

# eoscannonchn: HK, Hong Kong 
p2p-peer-address = node1.eoscannon.io:59876 

EDIT: And when i change chain-state-db-size-mb to smaller size like 3072, after start nodeos, watch top, nodeos's memory start to increase, and eventually killed by system. The output log looks like:

~/eos/build/programs/nodeos$ ./nodeos --config-dir mainnet_config/ --data-dir mainnet_data
2018-09-06T08:15:48.893 thread-0   chain_plugin.cpp:317          plugin_initialize    ] initializing chain plugin
Killed

[spartucus]

And it's reproducible, just use above config and command will do.

[spoonincode]

Can you please show output of ulimit -a on same console that you're launching nodeos from?

[spartucus]

Sure.

~/eos/build/programs/nodeos# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 42798
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 42798
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
~/eos/build/programs/node#

[spartucus]

Sorry I didn't metion it was Ubuntu16.04 x64, it is reproducible in Ubuntu18.04 too. What is werid is that I just tried at MacOS10.13.6, it's not reproduced.

[spoonincode]

Note that it is impossible to "shrink" the chain-state-db-size-mb once it has been created.

max locked memory (kbytes, -l) 64

I suspected this would be higher or unlimited, thus causing the mlock (fixed in 1.2.4) to be the cause of your issue. But with this limit the mlock should pretty much insta fail.

Could you do an ls -lR of your nodeos' data directory? I would like to see if anything in there is abnormally large (like the forkdb).

[spartucus]

@spoonincode Sorry for the late reply.

~/eos/build/programs/nodeos$ ls -lR
.:
total 386288
drwxrwxr-x 3 niubob niubob      4096 9月   6 16:18 CMakeFiles
-rw-rw-r-- 1 niubob niubob      3551 9月   6 16:18 cmake_install.cmake
drwxrwxr-x 2 niubob niubob      4096 8月  27 14:45 config
-rw-rw-r-- 1 niubob niubob       339 9月   6 15:11 config.hpp
-rw-rw-r-- 1 niubob niubob       283 7月  23 14:22 CTestTestfile.cmake
-rw-rw-r-- 1 niubob niubob         0 7月  23 17:52 fail_conn.txt
drwxrwxr-x 2 niubob niubob      4096 9月   6 18:09 mainnet_config
drwxrwxr-x 4 niubob niubob      4096 9月   6 18:05 mainnet_data
-rw-rw-r-- 1 niubob niubob      8058 7月  23 14:22 Makefile
-rwxrwxr-x 1 niubob niubob 395516720 9月   6 17:07 nodeos

./CMakeFiles:
total 12
-rw-rw-r-- 1 niubob niubob  613 7月  23 14:22 CMakeDirectoryInformation.cmake
drwxrwxr-x 2 niubob niubob 4096 9月   6 17:05 nodeos.dir
-rw-rw-r-- 1 niubob niubob    3 9月   6 16:18 progress.marks

./CMakeFiles/nodeos.dir:
total 4732
-rw-rw-r-- 1 niubob niubob   12463 9月   6 16:18 build.make
-rw-rw-r-- 1 niubob niubob     238 9月   6 16:18 cmake_clean.cmake
-rw-rw-r-- 1 niubob niubob  595399 9月   6 15:43 CXX.includecache
-rw-rw-r-- 1 niubob niubob    5544 8月  28 11:22 DependInfo.cmake
-rw-rw-r-- 1 niubob niubob  159388 9月   6 15:43 depend.internal
-rw-rw-r-- 1 niubob niubob  275750 9月   6 15:43 depend.make
-rw-rw-r-- 1 niubob niubob    2332 9月   6 16:18 flags.make
-rw-rw-r-- 1 niubob niubob    5545 9月   6 16:18 link.txt
-rw-rw-r-- 1 niubob niubob 3762928 9月   6 17:05 main.cpp.o
-rw-rw-r-- 1 niubob niubob      43 9月   6 16:18 progress.make

./config:
total 12
-rw-rw-r-- 1 niubob niubob 1125 8月  27 14:45 config.ini
-rw-rw-r-- 1 niubob niubob 2312 7月  31 10:58 config.ini.bak
-rw-rw-r-- 1 niubob niubob  816 8月   1 10:16 genesis.json

./mainnet_config:
total 8
-rw-rw-r-- 1 niubob niubob 2936 9月   6 18:09 config.ini
-rw-rw-r-- 1 niubob niubob  816 9月   6 15:04 genesis.json

./mainnet_data:
total 8
drwxrwxr-x 3 niubob niubob 4096 9月   6 18:05 blocks
drwxrwxr-x 2 niubob niubob 4096 9月  10 09:30 state

./mainnet_data/blocks:
total 12
-rw-rw-r-- 1 niubob niubob    8 9月   6 18:05 blocks.index
-rw-rw-r-- 1 niubob niubob  306 9月   6 18:05 blocks.log
drwxrwxr-x 2 niubob niubob 4096 9月   6 18:05 reversible

./mainnet_data/blocks/reversible:
total 12
-rw-rw-r-- 1 niubob niubob 524288000 9月  10 09:30 shared_memory.bin
-rw-rw-r-- 1 niubob niubob      2896 9月  10 09:30 shared_memory.meta

./mainnet_data/state:
total 5152
-rw-rw-r-- 1 niubob niubob         627 9月  10 09:30 forkdb.dat
-rw-rw-r-- 1 niubob niubob 32212254720 9月  10 09:30 shared_memory.bin
-rw-rw-r-- 1 niubob niubob        2896 9月  10 09:30 shared_memory.meta

[gs201510]

@spartucus Have you solved this?

[spartucus]

@gs201510 no, for now i'm using v1.2.3.

[spoonincode]

Since 1.2.5 doesn't mlock any more, I suspect this is resolved. Regardless, @spartucus were you running nodeos as root user when you encountered this problem?

[spartucus]

@spoonincode Yes, I was using sudo ./nodeos --config-dir ... command, since normal user have permission issue.

[spoonincode]

Closing this as fixed in 1.2.5, please follow up if not the case for you

[spartucus]

Yes, v1.2.5 works.