cc32d9
commented
3 years ago account query is enabled on all 4 nodes
Open cc32d9 opened 3 years ago
cc32d9
commented
3 years ago account query is enabled on all 4 nodes
cc32d9
commented
3 years ago Coredump here: https://snapshots.eosamsterdam.net/public/github_issues/10166/
I'm upgrading the system to 2.0.11 now.
cc32d9
commented
3 years ago I will also leave only one node with account queries and direct all account queries to it
cc32d9
commented
3 years ago two of four processes went above 100% usage, and account queries are turned off on them.
cc32d9
commented
3 years ago the configuration here:
chain-state-db-size-mb = 65536
reversible-blocks-db-size-mb = 2048
wasm-runtime = eos-vm-jit
eos-vm-oc-enable = true
validation-mode = light
http-server-address = 127.0.0.1:8802
http-max-response-time-ms = 100
abi-serializer-max-time-ms = 50
p2p-listen-endpoint = 0.0.0.0:9802
p2p-server-address = 95.217.42.35:9876
http-validate-host = false
access-control-allow-origin = *
access-control-allow-headers = Origin, X-Requested-With, Content-Type, Accept
verbose-http-errors = true
contracts-console = true
max-clients = 800
p2p-max-nodes-per-host = 800
plugin = eosio::chain_plugin
plugin = eosio::chain_api_plugin
#enable-account-queries = true
sync-fetch-span = 50
p2p-peer-address = 127.0.0.1:9801
p2p-peer-address = node2.zbeos.com:9876
p2p-peer-address = bp.cryptolions.io:9876the CPU usage doesn't go down after cutting off public API and p2p traffic
cc32d9
commented
3 years ago here's the log. Interestingly, long after it got the kill signal and HTTP interface became unavailable, HTTP plugin reported transaction errors. Also by that time the node was cut off from public traffic for few minutes:
root@eos02:/var/local# kill 21579
root@eos02:/var/local# journalctl -u nodeos@api02 -f
-- Logs begin at Thu 2021-03-18 13:57:32 UTC. --
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: boid-api-on-eos.animusystems.com:3232 - 1db8607
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: :9801 - dab035a
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: 46553b17cfc1:9876 - 8452b05
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: p2p.eos.eosargentina.io - c817d4d
Mar 22 08:05:26 eos02 nodeos[21579]: error 2021-03-22T08:05:26.745 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: 95.217.61.21:9876 - 6a8564f
Mar 22 08:05:56 eos02 nodeos[21579]: error 2021-03-22T08:05:56.847 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:06:26 eos02 nodeos[21579]: error 2021-03-22T08:06:26.960 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:06:57 eos02 nodeos[21579]: error 2021-03-22T08:06:57.062 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:07:27 eos02 nodeos[21579]: error 2021-03-22T08:07:27.191 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:07:57 eos02 nodeos[21579]: error 2021-03-22T08:07:57.293 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:08:27 eos02 nodeos[21579]: error 2021-03-22T08:08:27.428 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:08:57 eos02 nodeos[21579]: error 2021-03-22T08:08:57.529 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:09:27 eos02 nodeos[21579]: error 2021-03-22T08:09:27.631 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:09:57 eos02 nodeos[21579]: error 2021-03-22T08:09:57.758 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:10:27 eos02 nodeos[21579]: error 2021-03-22T08:10:27.859 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:10:57 eos02 nodeos[21579]: error 2021-03-22T08:10:57.961 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:11:28 eos02 nodeos[21579]: error 2021-03-22T08:11:28.053 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:11:28 eos02 nodeos[21579]: error 2021-03-22T08:11:28.062 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:11:58 eos02 nodeos[21579]: error 2021-03-22T08:11:58.164 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:12:19 eos02 nodeos[21579]: error 2021-03-22T08:12:19.872 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:12:28 eos02 nodeos[21579]: error 2021-03-22T08:12:28.266 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:12:58 eos02 nodeos[21579]: error 2021-03-22T08:12:58.367 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:13:26 eos02 nodeos[21579]: error 2021-03-22T08:13:26.636 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:13:27 eos02 nodeos[21579]: error 2021-03-22T08:13:27.258 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:13:28 eos02 nodeos[21579]: error 2021-03-22T08:13:28.468 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:13:58 eos02 nodeos[21579]: error 2021-03-22T08:13:58.570 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:14:28 eos02 nodeos[21579]: error 2021-03-22T08:14:28.671 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:14:34 eos02 nodeos[21579]: error 2021-03-22T08:14:34.736 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:14:43 eos02 nodeos[21579]: error 2021-03-22T08:14:43.361 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:14:58 eos02 nodeos[21579]: error 2021-03-22T08:14:58.773 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:15:25 eos02 nodeos[21579]: error 2021-03-22T08:15:25.086 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:15:28 eos02 nodeos[21579]: error 2021-03-22T08:15:28.874 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:15:58 eos02 nodeos[21579]: error 2021-03-22T08:15:58.980 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:16:29 eos02 nodeos[21579]: error 2021-03-22T08:16:29.081 net-0 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:16:59 eos02 nodeos[21579]: error 2021-03-22T08:16:59.183 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:17:25 eos02 nodeos[21579]: error 2021-03-22T08:17:25.976 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:26 eos02 nodeos[21579]: error 2021-03-22T08:17:26.669 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:29 eos02 nodeos[21579]: error 2021-03-22T08:17:29.285 net-1 net_plugin.cpp:2437 operator() ] Closing connection to: bp.cryptolions.io:9876
Mar 22 08:17:29 eos02 nodeos[21579]: error 2021-03-22T08:17:29.641 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:35 eos02 nodeos[21579]: error 2021-03-22T08:17:35.991 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:36 eos02 nodeos[21579]: error 2021-03-22T08:17:36.257 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:37 eos02 nodeos[21579]: error 2021-03-22T08:17:37.588 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:37 eos02 nodeos[21579]: error 2021-03-22T08:17:37.614 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.send_transaction
Mar 22 08:17:38 eos02 nodeos[21579]: error 2021-03-22T08:17:38.751 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:40 eos02 nodeos[21579]: error 2021-03-22T08:17:40.067 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:40 eos02 nodeos[21579]: error 2021-03-22T08:17:40.785 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:42 eos02 nodeos[21579]: error 2021-03-22T08:17:42.083 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:44 eos02 nodeos[21579]: error 2021-03-22T08:17:44.100 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:44 eos02 nodeos[21579]: error 2021-03-22T08:17:44.380 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:45 eos02 nodeos[21579]: error 2021-03-22T08:17:45.502 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:46 eos02 nodeos[21579]: error 2021-03-22T08:17:46.110 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:46 eos02 nodeos[21579]: error 2021-03-22T08:17:46.642 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:47 eos02 nodeos[21579]: error 2021-03-22T08:17:47.766 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:49 eos02 nodeos[21579]: error 2021-03-22T08:17:49.315 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:50 eos02 nodeos[21579]: error 2021-03-22T08:17:50.252 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:51 eos02 nodeos[21579]: error 2021-03-22T08:17:51.490 nodeos http_plugin.cpp:964 handle_exception ] FC Exception encountered while processing chain.push_transaction
Mar 22 08:17:58 eos02 nodeos[21579]: corrupted size vs. prev_size
Mar 22 08:18:03 eos02 systemd[1]: nodeos@api02.service: Main process exited, code=dumped, status=6/ABRT
Mar 22 08:18:03 eos02 systemd[1]: nodeos@api02.service: Failed with result 'core-dump'.a new coredump is uploaded, this time with eosio_2.0.11-1-ubuntu-18.04_amd64.deb from github
cc32d9
commented
3 years ago one node went into this state again, after about a day of normal work. This time I redirected the traffic differently, so this node was not exposed to HTTP requests, and had only its p2p open to public. There's no extreme number of incoming p2p sessions, about 30 in total. The other process in similar configuration is still running fine.
cc32d9
commented
3 years ago I cut off the public p2p traffic from affected node, and few minutes later its CPU usage went back to normal, and I could stop it gracefully.
cc32d9
commented
3 years ago I configured disable-subjective-billing = true today, and now one of processes is at over 100% again. I will try to stop it later today
matthewdarwin
commented
3 years ago @cc32d9 is this still an issue? or subjective billing fixed everything?
cc32d9
commented
3 years ago the CPU load is very moderate since subjective billing was enabled on p2p.
One outstanding problem is still the shutdown procedure: when the transaction queue was full, the node could not stop gracefully.
EOS mainnet public node, running 2.0.10 patched deb provided by B1. The CPU usage goes eventually over 100%, while nodes continue responding on HTTP queries, but slower than usual.
Trying to stop the node with "kill", the node keeps running for few minutes, then coredumps.
systemd log:
coredump file is available, I'm uploading it now.