High CPU load and RAM

[HVNC1]

Environment

• Smapp version: 1.0.14
• OS: Windows 10

Describe the bug

high consumption of RAM and processor during node operation spacemesh-log-7c8cef2b.zip

[HVNC1]

spacemesh-log-7c8cef2b.zip

[dshulyak]

we will need to collect profiles to understand. if someone has a golang tooling installed on a system, please upload highlighed profiles below

1. needs to be added to config

"main": {
        "pprof-server": true
 }

2. heap

dd@dd-System-Product-Name:~/genesis$ go tool pprof http://localhost:6060/debug/pprof/heap Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap Saved profile in /home/dd/pprof/pprof.go-spacemesh.alloc_objects.alloc_space.inuse_objects.inuse_space.118.pb.gz File: go-spacemesh Build ID: 678e5d393ec5ec0a1259eaffc11f9e199ba9a625 Type: inuse_space Time: Aug 12, 2023 at 5:52pm (CEST) Entering interactive mode (type "help" for commands, "o" for options) (pprof)

3. cpu trace

dd@dd-System-Product-Name:~/genesis$ go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=30 Saved profile in /home/dd/pprof/pprof.go-spacemesh.samples.cpu.025.pb.gz File: go-spacemesh Build ID: 678e5d393ec5ec0a1259eaffc11f9e199ba9a625 Type: cpu Time: Aug 12, 2023 at 5:52pm (CEST) Duration: 30s, Total samples = 190ms ( 0.63%) Entering interactive mode (type "help" for commands, "o" for options)

[HVNC1]

After these errors, RAM starts to grow

[scj85]

Same here:

1) Spacemesh process is creating a new post data 2) Vmmem process - Spacemesh in Hyper-V with is running another node that smeshing.

Both consume around 30-40% of CPU (sometimes even more for short while). My processor is i9-10850K.

When syncing:

Logs: spacemesh-log-7c8cef2b.zip

[scj85]

Seems to be you have Guys infinity loop somewhere:

...
2023-08-12T23:56:09.961+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}
2023-08-12T23:56:09.961+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}
2023-08-12T23:56:09.961+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}
2023-08-12T23:56:09.961+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}
2023-08-12T23:56:09.961+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}
2023-08-12T23:56:09.970+0100    INFO    e2d27.mesh  consensus results   {"node_id": "e2d279e52ded8a8b4f8d229e04b3e64767f7895e64c866bdeb9fdfb1154645a6", "module": "mesh", "sessionId": "00bf6ae1-c085-4153-9fef-c008acdaa608", "layer_id": 8529, "results": [{"layer": 8520, "opinion": "f35da5d25b", "verified": true, "blocks": [{"id": "7e1c7ef71e", "layer": 8520, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8521, "opinion": "a0b3c73b29", "verified": false, "blocks": [{"id": "db387f8a3c", "layer": 8521, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8522, "opinion": "382ee9a592", "verified": false, "blocks": [{"id": "7628c801d7", "layer": 8522, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8523, "opinion": "1e755e5b2b", "verified": false, "blocks": [{"id": "f26f52c326", "layer": 8523, "height": 0, "valid": true, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8524, "opinion": "563c3e7df0", "verified": false, "blocks": [{"id": "1cd19b50df", "layer": 8524, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": true}]}, {"layer": 8525, "opinion": "c4b40ec387", "verified": false, "blocks": [{"id": "2425021d39", "layer": 8525, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8526, "opinion": "6333cbf659", "verified": false, "blocks": [{"id": "a99643a1cf", "layer": 8526, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8527, "opinion": "4098469b6e", "verified": false, "blocks": [{"id": "951d6f20e8", "layer": 8527, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8528, "opinion": "b58f1433ad", "verified": false, "blocks": [{"id": "254024d8c5", "layer": 8528, "height": 0, "valid": false, "invalid": false, "hare": false, "data": false, "local": true}]}, {"layer": 8529, "opinion": "e4f96185e0", "verified": false, "blocks": [{"id": "3dbd971206", "layer": 8529, "height": 0, "valid": false, "invalid": false, "hare": true, "data": true, "local": false}]}], "name": "mesh"}

...

Every ms are adding multiple of these messages.

PS: When I use only "go-spacemesh.exe", all looks good. No high CPU load issue.

[HVNC1]

I found a workaround while the developers work on it https://medium.com/@wnft__chia/how-to-run-spacemesh-using-the-command-line-7f1f86a13cf4

[haverlinkfor]

have the same problem after restarting the app. Any news from developers concerning this problem ?

[dshulyak]

@haverlinkfor which version?

[scj85]

@haverlinkfor which version?

On my side, CPU and RAM consumption is still very high (smapp v1.0.15):

[knopki]

@dshulyak v1.0.17 profiles 2023-08-14-pprof.go-spacemesh.tar.gz

[dshulyak]

thank you, implementation one of the API methods is not "well" considered

[speaker365]

Environment Smapp version: 1.0.15 OS: Ubuntu 22.04 Describe the bug high consumption of RAM and processor during node operation Before the update, I was on version 1.0.14 and this was not observed

[dshulyak]

profiles can be collected without go tool, such as:

wget -O heap.pprof http://0.0.0.0:6060/debug/pprof/heap wget -O cpu.pprof http://0.0.0.0:6060/debug/pprof/profile?seconds=30

[Odino1978]

I'm already on version 1.3.1 , all the machines are updated

[pigmej]

Anyone who encounters that issue please.

1. add "pprof-server": true, to the config at the main level
2. restart the node
3. wait till it reaches the high memory state and open in any of your browsers:
   • http://127.0.0.1:6060/debug/pprof/profile?seconds=30
   • http://127.0.0.1:6060/debug/pprof/heap

Upload these two files here.