Open cc32d9 opened 3 years ago
account query is enabled on all 4 nodes
Coredump here: https://snapshots.eosamsterdam.net/public/github_issues/10166/
I'm upgrading the system to 2.0.11 now.
I will also leave only one node with account queries and direct all account queries to it
two of four processes went above 100% usage, and account queries are turned off on them.
the configuration here:
chain-state-db-size-mb = 65536
reversible-blocks-db-size-mb = 2048
wasm-runtime = eos-vm-jit
eos-vm-oc-enable = true
validation-mode = light
http-server-address = 127.0.0.1:8802
http-max-response-time-ms = 100
abi-serializer-max-time-ms = 50
p2p-listen-endpoint = 0.0.0.0:9802
p2p-server-address = 95.217.42.35:9876
http-validate-host = false
access-control-allow-origin = *
access-control-allow-headers = Origin, X-Requested-With, Content-Type, Accept
verbose-http-errors = true
contracts-console = true
max-clients = 800
p2p-max-nodes-per-host = 800
plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
#enable-account-queries = true
sync-fetch-span = 50
p2p-peer-address = 127.0.0.1:9801
p2p-peer-address = node2.zbeos.com:9876
p2p-peer-address = bp.cryptolions.io:9876
the CPU usage doesn't go down after cutting off public API and p2p traffic
here's the log. Interestingly, long after it got the kill signal and HTTP interface became unavailable, HTTP plugin reported transaction errors. Also by that time the node was cut off from public traffic for few minutes:
root@eos02:/var/local# kill 21579
root@eos02:/var/local# journalctl -u nodeos@api02 -f
-- Logs begin at Thu 2021-03-18 13:57:32 UTC. --
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: boid-api-on-eos.animusystems.com:3232 - 1db8607
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: :9801 - dab035a
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: 46553b17cfc1:9876 - 8452b05
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: p2p.eos.eosargentina.io - c817d4d
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: 95.217.61.21:9876 - 6a8564f
Mar 22 08:05:56 eos02 nodeos[21579]: error 2021-03-22T08:05:56.847 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:06:26 eos02 nodeos[21579]: error 2021-03-22T08:06:26.960 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:06:57 eos02 nodeos[21579]: error 2021-03-22T08:06:57.062 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:07:27 eos02 nodeos[21579]: error 2021-03-22T08:07:27.191 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:07:57 eos02 nodeos[21579]: error 2021-03-22T08:07:57.293 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:08:27 eos02 nodeos[21579]: error 2021-03-22T08:08:27.428 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:08:57 eos02 nodeos[21579]: error 2021-03-22T08:08:57.529 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:09:27 eos02 nodeos[21579]: error 2021-03-22T08:09:27.631 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:09:57 eos02 nodeos[21579]: error 2021-03-22T08:09:57.758 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:10:27 eos02 nodeos[21579]: error 2021-03-22T08:10:27.859 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:10:57 eos02 nodeos[21579]: error 2021-03-22T08:10:57.961 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:11:28 eos02 nodeos[21579]: error 2021-03-22T08:11:28.053 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:11:28 eos02 nodeos[21579]: error 2021-03-22T08:11:28.062 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:11:58 eos02 nodeos[21579]: error 2021-03-22T08:11:58.164 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:12:19 eos02 nodeos[21579]: error 2021-03-22T08:12:19.872 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:12:28 eos02 nodeos[21579]: error 2021-03-22T08:12:28.266 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:12:58 eos02 nodeos[21579]: error 2021-03-22T08:12:58.367 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:13:26 eos02 nodeos[21579]: error 2021-03-22T08:13:26.636 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:13:27 eos02 nodeos[21579]: error 2021-03-22T08:13:27.258 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:13:28 eos02 nodeos[21579]: error 2021-03-22T08:13:28.468 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:13:58 eos02 nodeos[21579]: error 2021-03-22T08:13:58.570 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:14:28 eos02 nodeos[21579]: error 2021-03-22T08:14:28.671 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:14:34 eos02 nodeos[21579]: error 2021-03-22T08:14:34.736 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:14:43 eos02 nodeos[21579]: error 2021-03-22T08:14:43.361 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:14:58 eos02 nodeos[21579]: error 2021-03-22T08:14:58.773 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:15:25 eos02 nodeos[21579]: error 2021-03-22T08:15:25.086 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:15:28 eos02 nodeos[21579]: error 2021-03-22T08:15:28.874 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:15:58 eos02 nodeos[21579]: error 2021-03-22T08:15:58.980 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:16:29 eos02 nodeos[21579]: error 2021-03-22T08:16:29.081 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:16:59 eos02 nodeos[21579]: error 2021-03-22T08:16:59.183 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:17:25 eos02 nodeos[21579]: error 2021-03-22T08:17:25.976 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:26 eos02 nodeos[21579]: error 2021-03-22T08:17:26.669 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:29 eos02 nodeos[21579]: error 2021-03-22T08:17:29.285 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:17:29 eos02 nodeos[21579]: error 2021-03-22T08:17:29.641 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:35 eos02 nodeos[21579]: error 2021-03-22T08:17:35.991 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:36 eos02 nodeos[21579]: error 2021-03-22T08:17:36.257 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:37 eos02 nodeos[21579]: error 2021-03-22T08:17:37.588 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:37 eos02 nodeos[21579]: error 2021-03-22T08:17:37.614 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.send_transaction
Mar 22 08:17:38 eos02 nodeos[21579]: error 2021-03-22T08:17:38.751 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:40 eos02 nodeos[21579]: error 2021-03-22T08:17:40.067 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:40 eos02 nodeos[21579]: error 2021-03-22T08:17:40.785 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:42 eos02 nodeos[21579]: error 2021-03-22T08:17:42.083 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:44 eos02 nodeos[21579]: error 2021-03-22T08:17:44.100 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:44 eos02 nodeos[21579]: error 2021-03-22T08:17:44.380 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:45 eos02 nodeos[21579]: error 2021-03-22T08:17:45.502 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:46 eos02 nodeos[21579]: error 2021-03-22T08:17:46.110 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:46 eos02 nodeos[21579]: error 2021-03-22T08:17:46.642 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:47 eos02 nodeos[21579]: error 2021-03-22T08:17:47.766 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:49 eos02 nodeos[21579]: error 2021-03-22T08:17:49.315 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:50 eos02 nodeos[21579]: error 2021-03-22T08:17:50.252 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:51 eos02 nodeos[21579]: error 2021-03-22T08:17:51.490 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:58 eos02 nodeos[21579]: corrupted size vs. prev_size
Mar 22 08:18:03 eos02 systemd[1]: nodeos@api02.service: Main process exited, code=dumped, status=6/ABRT
Mar 22 08:18:03 eos02 systemd[1]: nodeos@api02.service: Failed with result 'core-dump'.
a new coredump is uploaded, this time with eosio_2.0.11-1-ubuntu-18.04_amd64.deb
from github
one node went into this state again, after about a day of normal work. This time I redirected the traffic differently, so this node was not exposed to HTTP requests, and had only its p2p open to public. There's no extreme number of incoming p2p sessions, about 30 in total. The other process in similar configuration is still running fine.
I cut off the public p2p traffic from affected node, and few minutes later its CPU usage went back to normal, and I could stop it gracefully.
I configured disable-subjective-billing = true
today, and now one of processes is at over 100% again. I will try to stop it later today
@cc32d9 is this still an issue? or subjective billing fixed everything?
the CPU load is very moderate since subjective billing was enabled on p2p.
One outstanding problem is still the shutdown procedure: when the transaction queue was full, the node could not stop gracefully.
EOS mainnet public node, running 2.0.10 patched deb provided by B1. The CPU usage goes eventually over 100%, while nodes continue responding on HTTP queries, but slower than usual.
Trying to stop the node with "kill", the node keeps running for few minutes, then coredumps.
systemd log:
coredump file is available, I'm uploading it now.