Closed pr0logas closed 4 years ago
What does the end of nodeos.log
look like?
Same as:
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-04T11:12:25.714 nodeos wabt.hpp:630 translate_one ] misaligned reference
And keep flushing the same message.
Update: For no reason ports are up now (without restart)
root@nodes-02 ~ # netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 4882/redis-server 1
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 25138/nodeos
tcp 0 0 0.0.0.0:9876 0.0.0.0:* LISTEN 25138/nodeos
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 659/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1125/sshd
tcp 0 0 0.0.0.0:7000 0.0.0.0:* LISTEN 27860/PM2 v3.5.1: G
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 25138/nodeos
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 1029/beam.smp
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 1038/zabbix_agentd
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 1029/beam.smp
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 29951/java
tcp6 0 0 :::4369 :::* LISTEN 1/init
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 29951/java
tcp6 0 0 :::22 :::* LISTEN 1125/sshd
tcp6 0 0 :::10050 :::* LISTEN 1038/zabbix_agentd
tcp6 0 0 :::5672 :::* LISTEN 1029/beam.smp
udp 0 0 127.0.0.53:53 0.0.0.0:* 659/systemd-resolve
nodeos.log still logging warn:
tail -n 50 nodeos.log
warn 2019-11-05T11:23:58.336 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.341 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.395 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.395 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.412 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.412 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.412 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.412 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.457 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.457 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.457 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.457 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.459 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.460 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.460 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.460 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-05T11:23:58.460 nodeos wabt.hpp:630 translate_one ] misaligned reference
This points to a bug in the code.. Flags:
--state-history-endpoint='127.0.0.1:8080' \
--http-server-address='0.0.0.0:8888' \
are not stable and turns on/off during syncing.
Update 2, not restarted yet:
Synced about ~1M Up to 34,946,049 Block and nodeos stalled. Syncing stopped. History plugin port and http server port open, but not responding:
root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 25138/nodeos
tcp 0 0 0.0.0.0:9876 0.0.0.0:* LISTEN 25138/nodeos
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 25138/nodeos
root@nodes-02 ~ # pgrep nodeos
25138
root@nodes-02 ~ # curl -m 10 http://localhost:8888/v1/chain/get_info -vv
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /v1/chain/get_info HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.58.0
> Accept: */*
>
* Operation timed out after 10001 milliseconds with 0 bytes received
* stopped the pause stream!
* Closing connection 0
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
nodeos.log stalled >
warn 2019-11-06T04:23:39.077 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.077 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.097 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.097 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.097 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.097 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.097 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.106 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.106 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-06T04:23:39.106 nodeos wabt.hpp:630 translate_one ] misaligned reference
What's happening with daemon is it v1.85 bug?
Are you querying state_history_plugin while it is syncing?
mostly, yes. There are npm jobs which trying to get the data from port 8080
If you have a large number of state_history_plugin requests it can prevent syncing and get_info from working.
Alright, stopping npm history plugin parse. Restarting nodeos daemon to see if it's helping to resume syncing without history plugin parsing.
Let us know how it goes.
root@nodes-02 ~ # pgrep nodeos
11431
Port 8888 and 8080 not working again
root@nodes-02 ~ # netstat -tulpn | grep nodeos
root@nodes-02 ~ #
nodeos.log flushing warn non-stop.
root@nodes-02 ~ # tail -f nodeos.log
warn 2019-11-07T08:50:31.086 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.089 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.089 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.091 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.094 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.098 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.098 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.107 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.112 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.118 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.125 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.127 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.132 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.135 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T08:50:31.138 nodeos wabt.hpp:630 translate_one ] misaligned reference
Looks like calls to 8888 or 8080 port has nothing to do with nodeos stability.
Update, stopping nodeos daemon with SIGNTERM:
warn 2019-11-07T14:34:25.843 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.847 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.849 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.851 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.855 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.855 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.856 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.865 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.868 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.871 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:34:25.875 nodeos wabt.hpp:613 translate_one ] misaligned const reference
warn 2019-11-07T14:34:25.878 nodeos wabt.hpp:630 translate_one ] misaligned reference
info 2019-11-07T14:34:25.881 nodeos controller.cpp:468 replay ] 23745500 of 34954733
info 2019-11-07T14:34:25.881 nodeos controller.cpp:475 replay ] 23745499 irreversible blocks replayed
info 2019-11-07T14:34:25.882 nodeos controller.cpp:480 replay ] resetting fork database with new last irreversible block as the new root: 016a53dc510143c12f16847bc8a522a0d119a7b82909f305a76c013873aeb98c
info 2019-11-07T14:34:25.882 nodeos controller.cpp:511 replay ] replayed 23745499 blocks in 84751 seconds, 3.56916020716853399 ms/block
info 2019-11-07T14:34:25.883 nodeos chain_plugin.cpp:982 plugin_startup ] starting chain in read/write mode
info 2019-11-07T14:34:25.883 nodeos chain_plugin.cpp:986 plugin_startup ] Blockchain started; head block is #23745500, genesis timestamp is 2018-06-08T08:08:08.888
info 2019-11-07T14:34:26.416 nodeos main.cpp:141 main ] nodeos successfully exiting
Blockchain started; head block is #23745500 ????
Update: starting log:
info 2019-11-07T14:42:08.410 nodeos chain_plugin.cpp:556 plugin_initialize ] initializing chain plugin
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:408 operator() ] Support for builtin protocol feature '
PREACTIVATE_FEATURE' (with digest of '0ec7e080177b2c02b278d5088611686b49d739925a92d9bfcacd7fc6b74053bd') is enabled without activat
ion restrictions
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
ONLY_LINK_TO_EXISTING_PERMISSION' (with digest of '1a99a59d87e06e09ec5b028a9cbb7749b4a5ad8819004365d02dc4379a8b7241') is enabled wi
th preactivation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
FORWARD_SETCODE' (with digest of '2652f5f96006294109b3dd0bbde63693f55324af452b799ee137a81a905eed25') is enabled with preactivation
required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
REPLACE_DEFERRED' (with digest of 'ef43112c6543b88db2283a2e077278c315ae2c84719a8b25f25cc88565fbea99') is enabled with preactivation
required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
NO_DUPLICATE_DEFERRED_ID' (with digest of '4a90c00d55454dc5b059055ca213579c6ea856967712a56017487886a4d4cc0f') is enabled with preac
tivation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
RAM_RESTRICTIONS' (with digest of '4e7bf348da00a945489b2a681749eb56f5de00b900014e137ddae39f48f69d67') is enabled with preactivation
required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
DISALLOW_EMPTY_PRODUCER_SCHEDULE' (with digest of '68dcaa34c0517d19666e6b33add67351d8c5f69e999ca1e37931bc410a297428') is enabled wi
th preactivation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
ONLY_BILL_FIRST_AUTHORIZER' (with digest of '8ba52fe7a3956c5cd3a656a3174b931d3bb2abb45578befc59f283ecd816a405') is enabled with pre
activation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
RESTRICT_ACTION_TO_SELF' (with digest of 'ad9e3d8f650687709fd68f4b90b41f7d825a365b02c23a636cef88ac2ac00c43') is enabled with preact
ivation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
FIX_LINKAUTH_RESTRICTION' (with digest of 'e0fb64b1085cc5538970158d05a009c24e276fb94e1a0bf6a528b48fbc4ff526') is enabled with preac
tivation required
info 2019-11-07T14:42:08.415 nodeos chain_plugin.cpp:395 operator() ] Support for builtin protocol feature '
GET_SENDER' (with digest of 'f0af56d2c5a48d60a4a5b5c903edfb7db3a736a94ed589d0b797df33ff9d3e1d') is enabled with preactivation requi
red
info 2019-11-07T14:42:08.417 nodeos block_log.cpp:125 open ] Log is nonempty
info 2019-11-07T14:42:08.417 nodeos block_log.cpp:152 open ] Index is nonempty
info 2019-11-07T14:42:08.422 nodeos http_plugin.cpp:465 plugin_initialize ] configured http to listen on 0.0.0.0:8
888
info 2019-11-07T14:42:08.422 nodeos state_history_plugin.c:616 plugin_initialize ] ip_port: 127.0.0.1:8080 host: 127.0.0.
1 port: 8080
info 2019-11-07T14:42:08.424 nodeos state_history_log.hpp:217 open_log ] trace_history.log has blocks 2-2374550
0
info 2019-11-07T14:42:08.425 nodeos state_history_log.hpp:217 open_log ] chain_state_history.log has blocks 2-2
3745500
info 2019-11-07T14:42:08.426 nodeos http_plugin.cpp:412 operator() ] configured http with Access-Control-Al
low-Origin: *
info 2019-11-07T14:42:08.426 nodeos main.cpp:93 main ] nodeos version v1.8.5
info 2019-11-07T14:42:08.426 nodeos main.cpp:94 main ] nodeos using configuration file /root/
.local/share/eosio/nodeos/config/config.ini
info 2019-11-07T14:42:08.426 nodeos main.cpp:95 main ] nodeos data directory is /root/.local/
share/eosio/nodeos/data
info 2019-11-07T14:42:08.427 nodeos controller.cpp:463 replay ] existing block log, attempting to repl
ay from 23745501 to 34954733 blocks
info 2019-11-07T14:42:08.427 nodeos transaction_context.cp:103 deadline_timer ] Using 6us deadline timer for checktime
: min:1us max:17us mean:1us stddev:2us
warn 2019-11-07T14:42:08.518 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.551 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.551 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.594 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.596 nodeos wabt.hpp:613 translate_one ] misaligned const reference
warn 2019-11-07T14:42:08.600 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.604 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.606 nodeos wabt.hpp:613 translate_one ] misaligned const reference
warn 2019-11-07T14:42:08.618 nodeos wabt.hpp:613 translate_one ] misaligned const reference
warn 2019-11-07T14:42:08.622 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.622 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.629 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.632 nodeos wabt.hpp:630 translate_one ] misaligned reference
warn 2019-11-07T14:42:08.648 nodeos wabt.hpp:630 translate_one ] misaligned reference
Problem not solved not sure what to do?
@pr0logas - we're working on reproducing the issue. I'll report back here with our findings.
Ports aren't opened until after a replay is complete. That's why they aren't opened initially and then later do appear as opened. Unless you need contracts' console output I would disable --contracts-console
which is what is responsible for all the misaligned
output; that will make it easier for you to watch the log for useful info such as replay progress.
IT would be really nice if get_info
or similar was available while replay was running so we could remotely check on progress of a replay.
@pr0logas, I see where you mentioned you are using v1.8.5, but could you please show us the output of this command?
$ nodeos --full-version || nodeos --version
Thanks :)
Here you are: @kj4ezj root@nodes-02 ~ # nodeos --full-version || nodeos --version
error 2019-11-11T08:37:25.228 nodeos main.cpp:120 main ] Throw location unknown (consider using BOOST_THROW_EXCEPTION)
Dynamic exception type: boost::wrapexcept<boost::program_options::unknown_option>
std::exception::what: unrecognised option '--full-version'
v1.8.5
After 2 days without restart ports are up :)
root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 23552/nodeos
tcp 0 0 0.0.0.0:9876 0.0.0.0:* LISTEN 23552/nodeos
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 23552/nodeos ```
tail -n 50 nodeos.log
```info 2019-11-11T08:51:39.295 nodeos net_plugin.cpp:1872 operator() ] host: boot.eostitan.com port: 9876
info 2019-11-11T08:51:39.295 nodeos net_plugin.cpp:1872 operator() ] host: api-full1.eoseoul.io port: 9876
info 2019-11-11T08:51:39.295 nodeos net_plugin.cpp:1872 operator() ] host: bp.eosbeijing.one port: 8080
info 2019-11-11T08:51:39.295 nodeos net_plugin.cpp:1872 operator() ] host: bp.cryptolions.io port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: api-full2.eoseoul.io port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: bp.libertyblock.io port: 9800
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: br.eosrio.io port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: eos-seed-de.privex.io port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: boot.eostitan.com port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: api-full1.eoseoul.io port: 9876
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: bp.eosbeijing.one port: 8080
info 2019-11-11T08:52:09.295 nodeos net_plugin.cpp:1872 operator() ] host: bp.cryptolions.io port: 9876
But blocks not syncing we are stall again with a deadline.
I started a test node with your parameters at the beginning of the week, but it is still syncing. I will give you a more detailed update soon.
@kj4ezj In my case you should reach ~30M without any porblems
@kj4ezj any news?
@pr0logas Did you remove the console print of smart contracts?
@mauhcs Only right now. I had to kill nodeos with kill -9 because daemon not responded to SIGNTERM anyway. Started again with additional flags:
--hard-replay-blockchain \
--replay-blockchain \
Let's wait couple of hours till block recover:
info 2019-11-20T13:32:44.968 nodeos block_log.cpp:477 repair_log ] Recovered block 8207000
info 2019-11-20T13:32:45.027 nodeos block_log.cpp:477 repair_log ] Recovered block 8208000
info 2019-11-20T13:32:45.070 nodeos block_log.cpp:477 repair_log ] Recovered block 8209000
info 2019-11-20T13:32:45.120 nodeos block_log.cpp:477 repair_log ] Recovered block 8210000
info 2019-11-20T13:32:45.163 nodeos block_log.cpp:477 repair_log ] Recovered block 8211000
info 2019-11-20T13:32:45.221 nodeos block_log.cpp:477 repair_log ] Recovered block 8212000
info 2019-11-20T13:32:45.268 nodeos block_log.cpp:477 repair_log ] Recovered block 8213000
info 2019-11-20T13:32:45.322 nodeos block_log.cpp:477 repair_log ] Recovered block 8214000
info 2019-11-20T13:32:45.377 nodeos block_log.cpp:477 repair_log ] Recovered block 8215000
info 2019-11-20T13:32:45.423 nodeos block_log.cpp:477 repair_log ] Recovered block 8216000
info 2019-11-20T13:32:45.475 nodeos block_log.cpp:477 repair_log ] Recovered block 8217000
info 2019-11-20T13:32:45.528 nodeos block_log.cpp:477 repair_log ] Recovered block 8218000
Plus upgraded to version:
nodeos -v
v1.8.6
@mauhcs After DB recover nodeos started to sync blocks again. Unfortunately, our ~50M blocks somehow reduced to 17M and started from this point...
669000 @ 2018-09-16T05:55:11.000 signed by libertyblock [trxs: 5, lib: 16668665, conf: 0, latency: 37246893641 ms]
info 2019-11-21T08:16:47.047 nodeos net_plugin.cpp:1872 operator() ] host: boot.eostitan.com port: 9876
info 2019-11-21T08:16:47.047 nodeos net_plugin.cpp:1872 operator() ] host: eos-seed-de.privex.io port: 9876
info 2019-11-21T08:16:47.047 nodeos net_plugin.cpp:1872 operator() ] host: bp.libertyblock.io port: 9800
info 2019-11-21T08:16:47.047 nodeos net_plugin.cpp:1872 operator() ] host: bp.eosbeijing.one port: 8080
info 2019-11-21T08:16:47.047 nodeos net_plugin.cpp:1872 operator() ] host: api-full2.eoseoul.io port: 9876
info 2019-11-21T08:16:48.185 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block c3e3b1274b47ea7c... #16670000 @ 2018-09-16T06:03:31.000 signed by libertyblock [trxs: 2, lib: 16669673, conf: 0, latency: 37246397185 ms]
info 2019-11-21T08:16:51.638 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 29ff116cfce00760... #16671000 @ 2018-09-16T06:11:51.500 signed by jedaaaaaaaaa [trxs: 4, lib: 16670668, conf: 0, latency: 37245900138 ms]
info 2019-11-21T08:16:56.560 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block fadd54358b49b404... #16672000 @ 2018-09-16T06:20:11.500 signed by eosswedenorg [trxs: 2, lib: 16671664, conf: 0, latency: 37245405060 ms]
info 2019-11-21T08:17:00.187 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block f79bff535de7104f... #16673000 @ 2018-09-16T06:28:31.500 signed by eosswedenorg [trxs: 3, lib: 16672672, conf: 0, latency: 37244908687 ms]
info 2019-11-21T08:17:03.486 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block f687357b3012abfa... #16674000 @ 2018-09-16T06:36:51.500 signed by eospaceioeos [trxs: 5, lib: 16673668, conf: 0, latency: 37244411986 ms]
info 2019-11-21T08:17:06.501 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 658d671f9cb1e4d8... #16675000 @ 2018-09-16T06:45:11.500 signed by eosnewyorkio [trxs: 5, lib: 16674664, conf: 0, latency: 37243915001 ms]
info 2019-11-21T08:17:09.455 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 969d250e1555bbc7... #16676000 @ 2018-09-16T06:53:31.500 signed by eosnewyorkio [trxs: 3, lib: 16675672, conf: 0, latency: 37243417955 ms]
info 2019-11-21T08:17:15.970 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 3357e29750a1485a... #16677000 @ 2018-09-16T07:01:51.500 signed by eoslaomaocom [trxs: 1, lib: 16676668, conf: 0, latency: 37242924470 ms]
Indeed, disk usage dropped from 1.2TB to ~600GB... I have no idea how that happened.............
@mauhcs Syncing stalled again: Last log:
info 2019-11-21T16:42:02.443 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block bcb21788ef83290a... #21578000 @ 2018-10-14T16:05:46.000 signed by eoslaomaocom [trxs: 15, lib: 21577667, conf: 0, latency: 34821376443 ms]
info 2019-11-21T16:42:11.832 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 858787162a02b4ea... #21579000 @ 2018-10-14T16:14:06.000 signed by eoslaomaocom [trxs: 15, lib: 21578675, conf: 240, latency: 34820885832 ms]
info 2019-11-21T16:42:21.077 nodeos net_plugin.cpp:1872 operator() ] host: boot.eostitan.com port: 9876
info 2019-11-21T16:42:21.077 nodeos net_plugin.cpp:1872 operator() ] host: eos-seed-de.privex.io port: 9876
info 2019-11-21T16:42:21.077 nodeos net_plugin.cpp:1872 operator() ] host: bp.libertyblock.io port: 9800
info 2019-11-21T16:42:21.077 nodeos net_plugin.cpp:1872 operator() ] host: bp.eosbeijing.one port: 8080
info 2019-11-21T16:42:21.077 nodeos net_plugin.cpp:1872 operator() ] host: api-full2.eoseoul.io port: 9876
info 2019-11-21T16:42:21.656 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block bee0cd7b1e39c307... #21580000 @ 2018-10-14T16:22:27.000 signed by eosiosg11111 [trxs: 19, lib: 21579671, conf: 0, latency: 34820394656 ms]
info 2019-11-21T16:42:31.574 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 753c53ec9541639f... #21581000 @ 2018-10-14T16:30:49.000 signed by eosiosg11111 [trxs: 13, lib: 21580674, conf: 0, latency: 34819902574 ms]
info 2019-11-21T16:42:42.830 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 692631ccd48b09d7... #21582000 @ 2018-10-14T16:39:09.500 signed by eoshuobipool [trxs: 9, lib: 21581668, conf: 0, latency: 34819413330 ms]
info 2019-11-21T16:42:51.082 nodeos net_plugin.cpp:1872 operator() ] host: boot.eostitan.com port: 9876
info 2019-11-21T16:42:51.082 nodeos net_plugin.cpp:1872 operator() ] host: eos-seed-de.privex.io port: 9876
info 2019-11-21T16:42:51.082 nodeos net_plugin.cpp:1872 operator() ] host: bp.libertyblock.io port: 9800
info 2019-11-21T16:42:51.082 nodeos net_plugin.cpp:1872 operator() ] host: bp.eosbeijing.one port: 8080
info 2019-11-21T16:42:51.082 nodeos net_plugin.cpp:1872 operator() ] host: api-full2.eoseoul.io port: 9876
info 2019-11-21T16:42:52.830 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 296ffe36989f765f... #21583000 @ 2018-10-14T16:47:30.000 signed by eoshuobipool [trxs: 14, lib: 21582676, conf: 239, latency: 34818922830 ms]
info 2019-11-21T16:43:02.593 nodeos producer_plugin.cpp:410 on_incoming_block ] Received block 618fc17936639a78... #21584000 @ 2018-10-14T16:55:50.500 signed by eosgenblockp [trxs: 8, lib: 21583670, conf: 0, latency: 34818432093 ms]
root@nodes-02 ~ # netstat -tulpn | grep nodeos
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 29061/nodeos
tcp 1 0 0.0.0.0:9876 0.0.0.0:* LISTEN 29061/nodeos
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 29061/nodeos
root@nodes-02 ~ # curl -m 10 127.0.0.1:8888/v1/chain/get_info -v
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /v1/chain/get_info HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.58.0
> Accept: */*
>
* Operation timed out after 10001 milliseconds with 0 bytes received
* stopped the pause stream!
* Closing connection 0
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
I ran a sync node using commit 3c553db73864ea19458512b4669cf9942dc59f57
with the following command...
argv: ['nodeos',
'-d', '/mnt/data',
'--config-dir', '/mnt/data/config',
'-e',
'-p', 'eosio',
'--genesis-json', '/mnt/data/config/genesis.json',
'--plugin', 'eosio::producer_plugin',
'--plugin', 'eosio::chain_api_plugin',
'--plugin', 'eosio::http_plugin',
'--plugin', 'eosio::state_history_plugin',
'--filter-on', '*',
'--access-control-allow-origin', '*',
'--trace-history',
'--chain-state-history',
'--producer-threads', '16',
'--chain-threads', '16',
'--http-threads', '16',
'--net-threads', '16',
'--contracts-console',
'--validation-mode', 'full',
'--chain-state-db-size-mb', '65536',
'--disable-replay-opts',
'--sync-fetch-span', '5000',
'--state-history-endpoint', '127.0.0.1:8080',
'--http-server-address', '0.0.0.0:8888',
'--http-validate-host', 'false',
'--p2p-peer-address', '185.253.188.1:19876',
'--p2p-peer-address', 'api-full1.eoseoul.io:9876',
'--p2p-peer-address', 'api-full2.eoseoul.io:9876',
'--p2p-peer-address', 'boot.eostitan.com:9876',
'--p2p-peer-address', 'bp.cryptolions.io:9876',
'--p2p-peer-address', 'bp.eosbeijing.one:8080',
'--p2p-peer-address', 'bp.libertyblock.io:9800',
'--p2p-peer-address', 'br.eosrio.io:9876',
'--p2p-peer-address', 'eos-seed-de.privex.io:9876',
'--verbose-http-errors']
...which is different from yours in two ways. First, I assumed -e eosio
is a typo, which was supposed to be -e -p eosio
. This should not make a difference with the problem at hand. Second, I was not able to sync from these peers so I had to include additional (public, external) peers defined in the config file.
This node seemed to take longer to sync than normal, however at this point the node is up to block 63.7 million. I have a sidecar sitting next to the node polling /v1/chain/get_info
every 15 seconds, and it did fail to receive a response intermittently. This is a known-issue, as HTTP request are prioritized lower than receiving new blocks. However, what seemed wrong to me is that the requests did not time out. Instead, I got ECONNREFUSED
...
2019-12-12 09:50:47 Querying nodeos at http://localhost:8888.
2019-12-12 09:50:47 ERROR: getInfo() failed to GET http://localhost:8888/v1/chain/get_info!
...with this error object:
{ Error: connect ECONNREFUSED 127.0.0.1:8888
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 8888,
config:
{ url: 'http://localhost:8888/v1/chain/get_info',
method: 'get',
headers:
{ Accept: 'application/json, text/plain, */*',
'User-Agent': 'axios/0.19.0' },
transformRequest: [ [Function: transformRequest] ],
transformResponse: [ [Function: transformResponse] ],
timeout: 0,
adapter: [Function: httpAdapter],
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
validateStatus: [Function: validateStatus],
data: undefined },
request:
Writable {
_writableState:
WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: true,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
bufferedRequest: null,
lastBufferedRequest: null,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: true,
autoDestroy: false,
bufferedRequestCount: 0,
corkedRequestsFree: [Object] },
writable: true,
_events:
[Object: null prototype] {
response: [Function: handleResponse],
error: [Function: handleRequestError] },
_eventsCount: 2,
_maxListeners: undefined,
_options:
{ protocol: 'http:',
maxRedirects: 21,
maxBodyLength: 10485760,
path: '/v1/chain/get_info',
method: 'GET',
headers: [Object],
agent: undefined,
auth: undefined,
hostname: 'localhost',
port: '8888',
nativeProtocols: [Object],
pathname: '/v1/chain/get_info' },
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 0,
_requestBodyBuffers: [],
_onNativeResponse: [Function],
_currentRequest:
ClientRequest {
_events: [Object],
_eventsCount: 6,
_maxListeners: undefined,
output: [],
outputEncodings: [],
outputCallbacks: [],
outputSize: 0,
writable: true,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: [Socket],
connection: [Socket],
_header:
'GET /v1/chain/get_info HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.19.0\r\nHost: localhost:8888\r\nConnection: close\r\n\r\n',
_onPendingData: [Function: noopPendingOutput],
agent: [Agent],
socketPath: undefined,
timeout: undefined,
method: 'GET',
path: '/v1/chain/get_info',
_ended: false,
res: null,
aborted: undefined,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
_redirectable: [Circular],
[Symbol(isCorked)]: false,
[Symbol(outHeadersKey)]: [Object] },
_currentUrl: 'http://localhost:8888/v1/chain/get_info' },
response: undefined,
isAxiosError: true,
toJSON: [Function] }
@heifner and I are going to look at this more closely.
The ECONNREFUSED
issue I identified in my last comment was not related to nodeos, it was related to our tooling. I have synced past the problematic parts of the chain we identified (~ block 34 million for you, ~ block 64 million for me) twice with both v1.8.6 and v1.8.7, using your settings. My JavaScript program was able to query nodeos every 15 seconds and, though nodeos sometimes took a long time to respond due to internal prioritization, I saw no instances of ECONNREFUSED
or ports being closed. I am closing this because I am not able to reproduce your node's behavior. Thank you anyways for opening an issue.
Hello,
We used this command line config in order to sync EOS with mainnet:
After 32M blocks these settings no more working and we can't find any open port on the system:
However syncing works and disk usage increasing:
Any thoughts?