Regression tests on the node 6.3.0

mh-concordium commented 5 months ago

[x] Startup time at the highest block - Mainnet
[x] Observe memory consumption at startup - Mainnet
[x] Observe memory consumption at the highest block - Mainnet
[x] Catch up
[x] TPS test
[x] Catch up Mainnet
- [x] Windows
- [x] Ubuntu
- [x] Mac
[x] Catch up Testnet
- [x] Windows
- [x] Ubuntu
- [x] Mac

mh-concordium commented 4 months ago

Ubuntu logs, out of band catchup completed Mainnet:

Feb 08 14:58:55 mh-ubuntu concordium-mainnet-node-6.3.0[4800]: [INFO  concordium_node] Starting out of band catch-up
Feb 09 20:53:06 mh-ubuntu concordium-mainnet-node-6.3.0[4800]: [INFO  concordium_node] Completed out of band catch-up from https://catchup.mainnet.concordium.software/blocks.idx.
Ending block: a3cd8bf1569df580369d1d5a4f89a31d4187d3eaf37249c0cd1bb7ad88e0dae7

Testnet (Node was restarted once in between completing catch up):

Feb 13 17:11:25 mh-ubuntu concordium-testnet-node-6.3.0[6388]: [INFO  concordium_node] Starting out of band catch-up
Feb 13 17:11:25 mh-ubuntu concordium-testnet-node-6.3.0[6388]: [INFO  concordium_node] Starting out of band catch-up

Windows logs, out of band catchup completed Mainnet:

2024-02-10T02:22:43.982890500+01:00 INFO concordium_node - Completed out of band catch-up from https://catchup.mainnet.concordium.software/blocks.idx.
Ending block: a3cd8bf1569df580369d1d5a4f89a31d4187d3eaf37249c0cd1bb7ad88e0dae7

Testnet:

2024-02-12T11:24:43.549453700+01:00 INFO concordium_node - Starting out of band catch-up
2024-02-13T06:38:48.507382100+01:00 INFO concordium_node - Completed out of band catch-up from https://catchup.testnet.concordium.com/blocks.idx.

mh-concordium commented 4 months ago

Startup time on Mainnet at the highest block: 0:00:29 seconds. Startup time logs:

Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::utils] Starting up concordium_node version 6.3.0!
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::utils] Application data directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/data
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::utils] Application config directory: /var/lib/concordium-9dd9ca4d19e9393877d2c44b70f89acbfc0883c2243e5eeaecc0d1cd0503f478/config
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::utils] Network: enabled
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::utils] Log level: info
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::p2p::maintenance] My Node ID is f0eb5069219106fd
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::p2p::maintenance] Listening on 172.17.0.1:8879
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node] Starting consensus layer
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::plugins::consensus] Starting up the consensus thread
Feb 12  10:04:59    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node] Consensus layer started
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::consensus_ffi::ffi] Konsensus: Starting consensus operations.
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::grpc2::server] Starting GRPC V2 server listening on 0.0.0.0:20000
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node::consensus_ffi::ffi] Runner: Transaction purging thread started.
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node] Starting out of band catch-up
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node] Completed out of band catch-up from https://catchup.mainnet.concordium.software/blocks.idx.
Feb 12  10:05:28    mh-ubuntu   concordium-mainnet-node-6.3.0[37328]:   [INFO   concordium_node] Starting the P2P layer

mh-concordium commented 4 months ago

Observed memory consumption at startup on Mainnet using Ubuntu machine. The node caught up to the highest block before starting it again.

Memory consumption goes up to 1GB during the startup time but after a shorter period (within 10 min after it started) goes down to ~600MB which is expected.

Following is a screenshot in time at the point when the memory went down. After startup impact, memory is down to ~600 MB.

PID   USER      PR  NI    VIRT    RES    SHR   S  %CPU     %MEM     TIME+       COMMAND
2851 concord+  20   0  1146.3g   1.3g   0.7g   S  16.9       8.5    3:33.37    concordium-main

Concordium / concordium-node

Regression tests on the node 6.3.0 #1120

[x] TPS test