nodeos ignoring setting flags --state-history-endpoint --http-server-address

[pr0logas]

Hello,

We used this command line config in order to sync EOS with mainnet:

nodeos -e eosio \
--genesis-json='genesis.json' \
--plugin eosio::producer_plugin \
--plugin eosio::chain_api_plugin \
--plugin eosio::http_plugin \
--plugin eosio::state_history_plugin \
--filter-on="*" \
--access-control-allow-origin='*' \
--trace-history \
--chain-state-history \
--producer-threads='16' \
--chain-threads='16' \
--http-threads='16' \
--net-threads='16' \
--contracts-console \
--validation-mode=full \
--chain-state-db-size-mb=65536 \
--disable-replay-opts \
--sync-fetch-span='5000' \
--state-history-endpoint='127.0.0.1:8080' \
--http-server-address='0.0.0.0:8888' \
--http-validate-host=false \
--p2p-peer-address='185.253.188.1:19876' \
--p2p-peer-address='api-full1.eoseoul.io:9876' \
--p2p-peer-address='api-full2.eoseoul.io:9876' \
--p2p-peer-address='boot.eostitan.com:9876' \
--p2p-peer-address='bp.cryptolions.io:9876' \
--p2p-peer-address='bp.eosbeijing.one:8080' \
--p2p-peer-address='bp.libertyblock.io:9800' \
--p2p-peer-address='br.eosrio.io:9876' \
--p2p-peer-address='eos-seed-de.privex.io:9876' \
--verbose-http-errors >> nodeos.log 2>&1 &

After 32M blocks these settings no more working and we can't find any open port on the system:

root@nodes-02 ~ # cat /etc/issue
Ubuntu 18.04.3 LTS \n \l

root@nodes-02 ~ # netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      4882/redis-server 1 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      659/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1125/sshd           
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      1029/beam.smp       
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1038/zabbix_agentd  
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      1029/beam.smp       
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      1897/java           
tcp6       0      0 :::4369                 :::*                    LISTEN      1/init              
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      1897/java           
tcp6       0      0 :::22                   :::*                    LISTEN      1125/sshd           
tcp6       0      0 :::10050                :::*                    LISTEN      1038/zabbix_agentd  
tcp6       0      0 :::5672                 :::*                    LISTEN      1029/beam.smp       
udp        0      0 127.0.0.53:53           0.0.0.0:*                           659/systemd-resolve 

root@nodes-02 ~ # cat nodeos.log | head -50
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:556          plugin_initialize    ] initializing chain plugin
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:408          operator()           ] Support for builtin protocol feature 'PREACTIVATE_FEATURE' (with digest of '0ec7e080177b2c02b278d5088611686b49d739925a92d9bfcacd7fc6b74053bd') is enabled without activation restrictions
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'ONLY_LINK_TO_EXISTING_PERMISSION' (with digest of '1a99a59d87e06e09ec5b028a9cbb7749b4a5ad8819004365d02dc4379a8b7241') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'FORWARD_SETCODE' (with digest of '2652f5f96006294109b3dd0bbde63693f55324af452b799ee137a81a905eed25') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'REPLACE_DEFERRED' (with digest of 'ef43112c6543b88db2283a2e077278c315ae2c84719a8b25f25cc88565fbea99') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'NO_DUPLICATE_DEFERRED_ID' (with digest of '4a90c00d55454dc5b059055ca213579c6ea856967712a56017487886a4d4cc0f') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'RAM_RESTRICTIONS' (with digest of '4e7bf348da00a945489b2a681749eb56f5de00b900014e137ddae39f48f69d67') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'DISALLOW_EMPTY_PRODUCER_SCHEDULE' (with digest of '68dcaa34c0517d19666e6b33add67351d8c5f69e999ca1e37931bc410a297428') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'ONLY_BILL_FIRST_AUTHORIZER' (with digest of '8ba52fe7a3956c5cd3a656a3174b931d3bb2abb45578befc59f283ecd816a405') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'RESTRICT_ACTION_TO_SELF' (with digest of 'ad9e3d8f650687709fd68f4b90b41f7d825a365b02c23a636cef88ac2ac00c43') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'FIX_LINKAUTH_RESTRICTION' (with digest of 'e0fb64b1085cc5538970158d05a009c24e276fb94e1a0bf6a528b48fbc4ff526') is enabled with preactivation required
info  2019-11-04T11:12:25.601 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature 'GET_SENDER' (with digest of 'f0af56d2c5a48d60a4a5b5c903edfb7db3a736a94ed589d0b797df33ff9d3e1d') is enabled with preactivation required
info  2019-11-04T11:12:25.602 nodeos    block_log.cpp:125             open                 ] Log is nonempty
info  2019-11-04T11:12:25.602 nodeos    block_log.cpp:152             open                 ] Index is nonempty
info  2019-11-04T11:12:25.604 nodeos    http_plugin.cpp:465           plugin_initialize    ] configured http to listen on 0.0.0.0:8888
info  2019-11-04T11:12:25.604 nodeos    state_history_plugin.c:616    plugin_initialize    ] ip_port: 127.0.0.1:8080 host: 127.0.0.1 port: 8080 
info  2019-11-04T11:12:25.604 nodeos    state_history_log.hpp:217     open_log             ] trace_history.log has blocks 2-33257000
info  2019-11-04T11:12:25.604 nodeos    state_history_log.hpp:217     open_log             ] chain_state_history.log has blocks 2-33257000
info  2019-11-04T11:12:25.605 nodeos    http_plugin.cpp:412           operator()           ] configured http with Access-Control-Allow-Origin: *
info  2019-11-04T11:12:25.605 nodeos    main.cpp:93                   main                 ] nodeos version v1.8.5
info  2019-11-04T11:12:25.605 nodeos    main.cpp:94                   main                 ] nodeos using configuration file /root/.local/share/eosio/nodeos/config/config.ini
info  2019-11-04T11:12:25.605 nodeos    main.cpp:95                   main                 ] nodeos data directory is /root/.local/share/eosio/nodeos/data
info  2019-11-04T11:12:25.605 nodeos    controller.cpp:463            replay               ] existing block log, attempting to replay from 33257001 to 33574929 blocks
info  2019-11-04T11:12:25.605 nodeos    transaction_context.cp:103    deadline_timer       ] Using 6us deadline timer for checktime: min:1us max:16us mean:1us stddev:2us
warn  2019-11-04T11:12:25.713 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.713 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

However syncing works and disk usage increasing:

root@nodes-02 ~ # pgrep nodeos
8383

root@nodes-02 ~/.local/share/eosio/nodeos # du data/ | grep data | tail -1
656932128   data/
root@nodes-02 ~/.local/share/eosio/nodeos # du data/ | grep data | tail -1
656933088   data/

Any thoughts?

[spoonincode]

What does the end of nodeos.log look like?

[pr0logas]

Same as:

warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-04T11:12:25.714 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

And keep flushing the same message.

[pr0logas]

Update: For no reason ports are up now (without restart)

root@nodes-02 ~ # netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      4882/redis-server 1 
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      25138/nodeos        
tcp        0      0 0.0.0.0:9876            0.0.0.0:*               LISTEN      25138/nodeos        
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      659/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1125/sshd           
tcp        0      0 0.0.0.0:7000            0.0.0.0:*               LISTEN      27860/PM2 v3.5.1: G 
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      25138/nodeos        
tcp        0      0 0.0.0.0:15672           0.0.0.0:*               LISTEN      1029/beam.smp       
tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      1038/zabbix_agentd  
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN      1029/beam.smp       
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      29951/java          
tcp6       0      0 :::4369                 :::*                    LISTEN      1/init              
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      29951/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1125/sshd           
tcp6       0      0 :::10050                :::*                    LISTEN      1038/zabbix_agentd  
tcp6       0      0 :::5672                 :::*                    LISTEN      1029/beam.smp       
udp        0      0 127.0.0.53:53           0.0.0.0:*                           659/systemd-resolve 

nodeos.log still logging warn: tail -n 50 nodeos.log

warn  2019-11-05T11:23:58.336 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.341 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.395 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.395 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.412 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.412 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.412 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.412 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.457 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.457 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.457 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.457 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.459 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.460 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.460 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.460 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-05T11:23:58.460 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

This points to a bug in the code.. Flags:

--state-history-endpoint='127.0.0.1:8080' \
--http-server-address='0.0.0.0:8888' \

are not stable and turns on/off during syncing.

[pr0logas]

Update 2, not restarted yet:

Synced about ~1M Up to 34,946,049 Block and nodeos stalled. Syncing stopped. History plugin port and http server port open, but not responding:

root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      25138/nodeos        
tcp        0      0 0.0.0.0:9876            0.0.0.0:*               LISTEN      25138/nodeos        
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      25138/nodeos        

root@nodes-02 ~ # pgrep nodeos
25138

root@nodes-02 ~ # curl -m 10 http://localhost:8888/v1/chain/get_info -vv
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /v1/chain/get_info HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Operation timed out after 10001 milliseconds with 0 bytes received
* stopped the pause stream!
* Closing connection 0
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received

nodeos.log stalled >

warn  2019-11-06T04:23:39.077 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.077 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.097 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.097 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.097 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.097 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.097 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.106 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.106 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-06T04:23:39.106 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

What's happening with daemon is it v1.85 bug?

[heifner]

Are you querying state_history_plugin while it is syncing?

[pr0logas]

mostly, yes. There are npm jobs which trying to get the data from port 8080

[heifner]

If you have a large number of state_history_plugin requests it can prevent syncing and get_info from working.

[pr0logas]

Alright, stopping npm history plugin parse. Restarting nodeos daemon to see if it's helping to resume syncing without history plugin parsing.

[heifner]

Let us know how it goes.

[pr0logas]

root@nodes-02 ~ # pgrep nodeos
11431

Port 8888 and 8080 not working again

root@nodes-02 ~ # netstat -tulpn | grep nodeos
root@nodes-02 ~ # 

nodeos.log flushing warn non-stop.

root@nodes-02 ~ # tail -f nodeos.log
warn  2019-11-07T08:50:31.086 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.089 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.089 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.091 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.094 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.098 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.098 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.107 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.112 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.118 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.125 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.127 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.132 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.135 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T08:50:31.138 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

Looks like calls to 8888 or 8080 port has nothing to do with nodeos stability.

[pr0logas]

Update, stopping nodeos daemon with SIGNTERM:

warn  2019-11-07T14:34:25.843 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.847 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.849 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.851 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.855 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.855 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.856 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.865 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.868 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.871 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:34:25.875 nodeos    wabt.hpp:613                  translate_one        ] misaligned const reference
warn  2019-11-07T14:34:25.878 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
info  2019-11-07T14:34:25.881 nodeos    controller.cpp:468            replay               ] 23745500 of 34954733
info  2019-11-07T14:34:25.881 nodeos    controller.cpp:475            replay               ] 23745499 irreversible blocks replayed
info  2019-11-07T14:34:25.882 nodeos    controller.cpp:480            replay               ] resetting fork database with new last irreversible block as the new root: 016a53dc510143c12f16847bc8a522a0d119a7b82909f305a76c013873aeb98c
info  2019-11-07T14:34:25.882 nodeos    controller.cpp:511            replay               ] replayed 23745499 blocks in 84751 seconds, 3.56916020716853399 ms/block
info  2019-11-07T14:34:25.883 nodeos    chain_plugin.cpp:982          plugin_startup       ] starting chain in read/write mode
info  2019-11-07T14:34:25.883 nodeos    chain_plugin.cpp:986          plugin_startup       ] Blockchain started; head block is #23745500, genesis timestamp is 2018-06-08T08:08:08.888
info  2019-11-07T14:34:26.416 nodeos    main.cpp:141                  main                 ] nodeos successfully exiting

Blockchain started; head block is #23745500 ????

[pr0logas]

Update: starting log:

info  2019-11-07T14:42:08.410 nodeos    chain_plugin.cpp:556          plugin_initialize    ] initializing chain plugin
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:408          operator()           ] Support for builtin protocol feature '
PREACTIVATE_FEATURE' (with digest of '0ec7e080177b2c02b278d5088611686b49d739925a92d9bfcacd7fc6b74053bd') is enabled without activat
ion restrictions
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
ONLY_LINK_TO_EXISTING_PERMISSION' (with digest of '1a99a59d87e06e09ec5b028a9cbb7749b4a5ad8819004365d02dc4379a8b7241') is enabled wi
th preactivation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
FORWARD_SETCODE' (with digest of '2652f5f96006294109b3dd0bbde63693f55324af452b799ee137a81a905eed25') is enabled with preactivation 
required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
REPLACE_DEFERRED' (with digest of 'ef43112c6543b88db2283a2e077278c315ae2c84719a8b25f25cc88565fbea99') is enabled with preactivation
 required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
NO_DUPLICATE_DEFERRED_ID' (with digest of '4a90c00d55454dc5b059055ca213579c6ea856967712a56017487886a4d4cc0f') is enabled with preac
tivation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
RAM_RESTRICTIONS' (with digest of '4e7bf348da00a945489b2a681749eb56f5de00b900014e137ddae39f48f69d67') is enabled with preactivation
 required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
DISALLOW_EMPTY_PRODUCER_SCHEDULE' (with digest of '68dcaa34c0517d19666e6b33add67351d8c5f69e999ca1e37931bc410a297428') is enabled wi
th preactivation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
ONLY_BILL_FIRST_AUTHORIZER' (with digest of '8ba52fe7a3956c5cd3a656a3174b931d3bb2abb45578befc59f283ecd816a405') is enabled with pre
activation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
RESTRICT_ACTION_TO_SELF' (with digest of 'ad9e3d8f650687709fd68f4b90b41f7d825a365b02c23a636cef88ac2ac00c43') is enabled with preact
ivation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
FIX_LINKAUTH_RESTRICTION' (with digest of 'e0fb64b1085cc5538970158d05a009c24e276fb94e1a0bf6a528b48fbc4ff526') is enabled with preac
tivation required
info  2019-11-07T14:42:08.415 nodeos    chain_plugin.cpp:395          operator()           ] Support for builtin protocol feature '
GET_SENDER' (with digest of 'f0af56d2c5a48d60a4a5b5c903edfb7db3a736a94ed589d0b797df33ff9d3e1d') is enabled with preactivation requi
red
info  2019-11-07T14:42:08.417 nodeos    block_log.cpp:125             open                 ] Log is nonempty
info  2019-11-07T14:42:08.417 nodeos    block_log.cpp:152             open                 ] Index is nonempty
info  2019-11-07T14:42:08.422 nodeos    http_plugin.cpp:465           plugin_initialize    ] configured http to listen on 0.0.0.0:8
888
info  2019-11-07T14:42:08.422 nodeos    state_history_plugin.c:616    plugin_initialize    ] ip_port: 127.0.0.1:8080 host: 127.0.0.
1 port: 8080 
info  2019-11-07T14:42:08.424 nodeos    state_history_log.hpp:217     open_log             ] trace_history.log has blocks 2-2374550
0
info  2019-11-07T14:42:08.425 nodeos    state_history_log.hpp:217     open_log             ] chain_state_history.log has blocks 2-2
3745500
info  2019-11-07T14:42:08.426 nodeos    http_plugin.cpp:412           operator()           ] configured http with Access-Control-Al
low-Origin: *
info  2019-11-07T14:42:08.426 nodeos    main.cpp:93                   main                 ] nodeos version v1.8.5
info  2019-11-07T14:42:08.426 nodeos    main.cpp:94                   main                 ] nodeos using configuration file /root/
.local/share/eosio/nodeos/config/config.ini
info  2019-11-07T14:42:08.426 nodeos    main.cpp:95                   main                 ] nodeos data directory is /root/.local/
share/eosio/nodeos/data
info  2019-11-07T14:42:08.427 nodeos    controller.cpp:463            replay               ] existing block log, attempting to repl
ay from 23745501 to 34954733 blocks
info  2019-11-07T14:42:08.427 nodeos    transaction_context.cp:103    deadline_timer       ] Using 6us deadline timer for checktime
: min:1us max:17us mean:1us stddev:2us
warn  2019-11-07T14:42:08.518 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.551 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.551 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.594 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.596 nodeos    wabt.hpp:613                  translate_one        ] misaligned const reference
warn  2019-11-07T14:42:08.600 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.604 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.606 nodeos    wabt.hpp:613                  translate_one        ] misaligned const reference
warn  2019-11-07T14:42:08.618 nodeos    wabt.hpp:613                  translate_one        ] misaligned const reference
warn  2019-11-07T14:42:08.622 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.622 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.629 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.632 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference
warn  2019-11-07T14:42:08.648 nodeos    wabt.hpp:630                  translate_one        ] misaligned reference

[pr0logas]

Problem not solved not sure what to do?

[jerstern]

@pr0logas - we're working on reproducing the issue. I'll report back here with our findings.

[spoonincode]

Ports aren't opened until after a replay is complete. That's why they aren't opened initially and then later do appear as opened. Unless you need contracts' console output I would disable --contracts-console which is what is responsible for all the misaligned output; that will make it easier for you to watch the log for useful info such as replay progress.

[matthewdarwin]

IT would be really nice if get_info or similar was available while replay was running so we could remotely check on progress of a replay.

[kj4ezj]

@pr0logas, I see where you mentioned you are using v1.8.5, but could you please show us the output of this command?

$ nodeos --full-version || nodeos --version

Thanks :)

[pr0logas]

Here you are: @kj4ezj root@nodes-02 ~ # nodeos --full-version || nodeos --version

error 2019-11-11T08:37:25.228 nodeos    main.cpp:120                  main                 ] Throw location unknown (consider using BOOST_THROW_EXCEPTION)
Dynamic exception type: boost::wrapexcept<boost::program_options::unknown_option>
std::exception::what: unrecognised option '--full-version'

v1.8.5

[pr0logas]

After 2 days without restart ports are up :)

root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      23552/nodeos        
tcp        0      0 0.0.0.0:9876            0.0.0.0:*               LISTEN      23552/nodeos        
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      23552/nodeos ```

tail -n 50 nodeos.log

```info  2019-11-11T08:51:39.295 nodeos    net_plugin.cpp:1872           operator()           ] host: boot.eostitan.com port: 9876 
info  2019-11-11T08:51:39.295 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full1.eoseoul.io port: 9876 
info  2019-11-11T08:51:39.295 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.eosbeijing.one port: 8080 
info  2019-11-11T08:51:39.295 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.cryptolions.io port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full2.eoseoul.io port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.libertyblock.io port: 9800 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: br.eosrio.io port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: eos-seed-de.privex.io port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: boot.eostitan.com port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full1.eoseoul.io port: 9876 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.eosbeijing.one port: 8080 
info  2019-11-11T08:52:09.295 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.cryptolions.io port: 9876

But blocks not syncing we are stall again with a deadline.

[kj4ezj]

I started a test node with your parameters at the beginning of the week, but it is still syncing. I will give you a more detailed update soon.

[pr0logas]

@kj4ezj In my case you should reach ~30M without any porblems

[pr0logas]

@kj4ezj any news?

[mauhcs]

@pr0logas Did you remove the console print of smart contracts?

[pr0logas]

@mauhcs Only right now. I had to kill nodeos with kill -9 because daemon not responded to SIGNTERM anyway. Started again with additional flags:

--hard-replay-blockchain \
--replay-blockchain \

Let's wait couple of hours till block recover:

info  2019-11-20T13:32:44.968 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8207000
info  2019-11-20T13:32:45.027 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8208000
info  2019-11-20T13:32:45.070 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8209000
info  2019-11-20T13:32:45.120 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8210000
info  2019-11-20T13:32:45.163 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8211000
info  2019-11-20T13:32:45.221 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8212000
info  2019-11-20T13:32:45.268 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8213000
info  2019-11-20T13:32:45.322 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8214000
info  2019-11-20T13:32:45.377 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8215000
info  2019-11-20T13:32:45.423 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8216000
info  2019-11-20T13:32:45.475 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8217000
info  2019-11-20T13:32:45.528 nodeos    block_log.cpp:477             repair_log           ] Recovered block 8218000

Plus upgraded to version:

nodeos -v
v1.8.6

[pr0logas]

@mauhcs After DB recover nodeos started to sync blocks again. Unfortunately, our ~50M blocks somehow reduced to 17M and started from this point...

669000 @ 2018-09-16T05:55:11.000 signed by libertyblock [trxs: 5, lib: 16668665, conf: 0, latency: 37246893641 ms]
info  2019-11-21T08:16:47.047 nodeos    net_plugin.cpp:1872           operator()           ] host: boot.eostitan.com port: 9876 
info  2019-11-21T08:16:47.047 nodeos    net_plugin.cpp:1872           operator()           ] host: eos-seed-de.privex.io port: 9876 
info  2019-11-21T08:16:47.047 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.libertyblock.io port: 9800 
info  2019-11-21T08:16:47.047 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.eosbeijing.one port: 8080 
info  2019-11-21T08:16:47.047 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full2.eoseoul.io port: 9876 
info  2019-11-21T08:16:48.185 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block c3e3b1274b47ea7c... #16670000 @ 2018-09-16T06:03:31.000 signed by libertyblock [trxs: 2, lib: 16669673, conf: 0, latency: 37246397185 ms]
info  2019-11-21T08:16:51.638 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 29ff116cfce00760... #16671000 @ 2018-09-16T06:11:51.500 signed by jedaaaaaaaaa [trxs: 4, lib: 16670668, conf: 0, latency: 37245900138 ms]
info  2019-11-21T08:16:56.560 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block fadd54358b49b404... #16672000 @ 2018-09-16T06:20:11.500 signed by eosswedenorg [trxs: 2, lib: 16671664, conf: 0, latency: 37245405060 ms]
info  2019-11-21T08:17:00.187 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block f79bff535de7104f... #16673000 @ 2018-09-16T06:28:31.500 signed by eosswedenorg [trxs: 3, lib: 16672672, conf: 0, latency: 37244908687 ms]
info  2019-11-21T08:17:03.486 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block f687357b3012abfa... #16674000 @ 2018-09-16T06:36:51.500 signed by eospaceioeos [trxs: 5, lib: 16673668, conf: 0, latency: 37244411986 ms]
info  2019-11-21T08:17:06.501 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 658d671f9cb1e4d8... #16675000 @ 2018-09-16T06:45:11.500 signed by eosnewyorkio [trxs: 5, lib: 16674664, conf: 0, latency: 37243915001 ms]
info  2019-11-21T08:17:09.455 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 969d250e1555bbc7... #16676000 @ 2018-09-16T06:53:31.500 signed by eosnewyorkio [trxs: 3, lib: 16675672, conf: 0, latency: 37243417955 ms]
info  2019-11-21T08:17:15.970 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 3357e29750a1485a... #16677000 @ 2018-09-16T07:01:51.500 signed by eoslaomaocom [trxs: 1, lib: 16676668, conf: 0, latency: 37242924470 ms]

Indeed, disk usage dropped from 1.2TB to ~600GB... I have no idea how that happened.............

[pr0logas]

@mauhcs Syncing stalled again: Last log:

info  2019-11-21T16:42:02.443 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block bcb21788ef83290a... #21578000 @ 2018-10-14T16:05:46.000 signed by eoslaomaocom [trxs: 15, lib: 21577667, conf: 0, latency: 34821376443 ms]
info  2019-11-21T16:42:11.832 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 858787162a02b4ea... #21579000 @ 2018-10-14T16:14:06.000 signed by eoslaomaocom [trxs: 15, lib: 21578675, conf: 240, latency: 34820885832 ms]
info  2019-11-21T16:42:21.077 nodeos    net_plugin.cpp:1872           operator()           ] host: boot.eostitan.com port: 9876 
info  2019-11-21T16:42:21.077 nodeos    net_plugin.cpp:1872           operator()           ] host: eos-seed-de.privex.io port: 9876 
info  2019-11-21T16:42:21.077 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.libertyblock.io port: 9800 
info  2019-11-21T16:42:21.077 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.eosbeijing.one port: 8080 
info  2019-11-21T16:42:21.077 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full2.eoseoul.io port: 9876 
info  2019-11-21T16:42:21.656 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block bee0cd7b1e39c307... #21580000 @ 2018-10-14T16:22:27.000 signed by eosiosg11111 [trxs: 19, lib: 21579671, conf: 0, latency: 34820394656 ms]
info  2019-11-21T16:42:31.574 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 753c53ec9541639f... #21581000 @ 2018-10-14T16:30:49.000 signed by eosiosg11111 [trxs: 13, lib: 21580674, conf: 0, latency: 34819902574 ms]
info  2019-11-21T16:42:42.830 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 692631ccd48b09d7... #21582000 @ 2018-10-14T16:39:09.500 signed by eoshuobipool [trxs: 9, lib: 21581668, conf: 0, latency: 34819413330 ms]
info  2019-11-21T16:42:51.082 nodeos    net_plugin.cpp:1872           operator()           ] host: boot.eostitan.com port: 9876 
info  2019-11-21T16:42:51.082 nodeos    net_plugin.cpp:1872           operator()           ] host: eos-seed-de.privex.io port: 9876 
info  2019-11-21T16:42:51.082 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.libertyblock.io port: 9800 
info  2019-11-21T16:42:51.082 nodeos    net_plugin.cpp:1872           operator()           ] host: bp.eosbeijing.one port: 8080 
info  2019-11-21T16:42:51.082 nodeos    net_plugin.cpp:1872           operator()           ] host: api-full2.eoseoul.io port: 9876 
info  2019-11-21T16:42:52.830 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 296ffe36989f765f... #21583000 @ 2018-10-14T16:47:30.000 signed by eoshuobipool [trxs: 14, lib: 21582676, conf: 239, latency: 34818922830 ms]
info  2019-11-21T16:43:02.593 nodeos    producer_plugin.cpp:410       on_incoming_block    ] Received block 618fc17936639a78... #21584000 @ 2018-10-14T16:55:50.500 signed by eosgenblockp [trxs: 8, lib: 21583670, conf: 0, latency: 34818432093 ms]

root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      29061/nodeos        
tcp        1      0 0.0.0.0:9876            0.0.0.0:*               LISTEN      29061/nodeos        
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      29061/nodeos   

root@nodes-02 ~ # curl -m 10 127.0.0.1:8888/v1/chain/get_info -v
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /v1/chain/get_info HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Operation timed out after 10001 milliseconds with 0 bytes received
* stopped the pause stream!
* Closing connection 0
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received

[kj4ezj]

I ran a sync node using commit 3c553db73864ea19458512b4669cf9942dc59f57 with the following command...

argv: ['nodeos',
'-d', '/mnt/data',
'--config-dir', '/mnt/data/config',
'-e',
'-p', 'eosio',
'--genesis-json', '/mnt/data/config/genesis.json',
'--plugin', 'eosio::producer_plugin',
'--plugin', 'eosio::chain_api_plugin',
'--plugin', 'eosio::http_plugin',
'--plugin', 'eosio::state_history_plugin',
'--filter-on', '*',
'--access-control-allow-origin', '*',
'--trace-history',
'--chain-state-history',
'--producer-threads', '16',
'--chain-threads', '16',
'--http-threads', '16',
'--net-threads', '16',
'--contracts-console',
'--validation-mode', 'full',
'--chain-state-db-size-mb', '65536',
'--disable-replay-opts',
'--sync-fetch-span', '5000',
'--state-history-endpoint', '127.0.0.1:8080',
'--http-server-address', '0.0.0.0:8888',
'--http-validate-host', 'false',
'--p2p-peer-address', '185.253.188.1:19876',
'--p2p-peer-address', 'api-full1.eoseoul.io:9876',
'--p2p-peer-address', 'api-full2.eoseoul.io:9876',
'--p2p-peer-address', 'boot.eostitan.com:9876',
'--p2p-peer-address', 'bp.cryptolions.io:9876',
'--p2p-peer-address', 'bp.eosbeijing.one:8080',
'--p2p-peer-address', 'bp.libertyblock.io:9800',
'--p2p-peer-address', 'br.eosrio.io:9876',
'--p2p-peer-address', 'eos-seed-de.privex.io:9876',
'--verbose-http-errors']

...which is different from yours in two ways. First, I assumed -e eosio is a typo, which was supposed to be -e -p eosio . This should not make a difference with the problem at hand. Second, I was not able to sync from these peers so I had to include additional (public, external) peers defined in the config file.

[kj4ezj]

This node seemed to take longer to sync than normal, however at this point the node is up to block 63.7 million. I have a sidecar sitting next to the node polling /v1/chain/get_info every 15 seconds, and it did fail to receive a response intermittently. This is a known-issue, as HTTP request are prioritized lower than receiving new blocks. However, what seemed wrong to me is that the requests did not time out. Instead, I got ECONNREFUSED...

2019-12-12 09:50:47 Querying nodeos at http://localhost:8888.
2019-12-12 09:50:47 ERROR: getInfo() failed to GET http://localhost:8888/v1/chain/get_info!

...with this error object:

{ Error: connect ECONNREFUSED 127.0.0.1:8888
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 8888,
  config:
   { url: 'http://localhost:8888/v1/chain/get_info',
     method: 'get',
     headers:
      { Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.19.0' },
     transformRequest: [ [Function: transformRequest] ],
     transformResponse: [ [Function: transformResponse] ],
     timeout: 0,
     adapter: [Function: httpAdapter],
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     data: undefined },
  request:
   Writable {
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        emitClose: true,
        autoDestroy: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     _events:
      [Object: null prototype] {
        response: [Function: handleResponse],
        error: [Function: handleRequestError] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options:
      { protocol: 'http:',
        maxRedirects: 21,
        maxBodyLength: 10485760,
        path: '/v1/chain/get_info',
        method: 'GET',
        headers: [Object],
        agent: undefined,
        auth: undefined,
        hostname: 'localhost',
        port: '8888',
        nativeProtocols: [Object],
        pathname: '/v1/chain/get_info' },
     _redirectCount: 0,
     _redirects: [],
     _requestBodyLength: 0,
     _requestBodyBuffers: [],
     _onNativeResponse: [Function],
     _currentRequest:
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [Socket],
        connection: [Socket],
        _header:
         'GET /v1/chain/get_info HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.19.0\r\nHost: localhost:8888\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path: '/v1/chain/get_info',
        _ended: false,
        res: null,
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl: 'http://localhost:8888/v1/chain/get_info' },
  response: undefined,
  isAxiosError: true,
  toJSON: [Function] }

@heifner and I are going to look at this more closely.

[kj4ezj]

The ECONNREFUSED issue I identified in my last comment was not related to nodeos, it was related to our tooling. I have synced past the problematic parts of the chain we identified (~ block 34 million for you, ~ block 64 million for me) twice with both v1.8.6 and v1.8.7, using your settings. My JavaScript program was able to query nodeos every 15 seconds and, though nodeos sometimes took a long time to respond due to internal prioritization, I saw no instances of ECONNREFUSED or ports being closed. I am closing this because I am not able to reproduce your node's behavior. Thank you anyways for opening an issue.