ton-blockchain / ton

Main TON monorepo
Other
3.1k stars 982 forks source link

Ton validator 2024-09 and 10 has major memory leak #1301

Open astudnev opened 1 month ago

astudnev commented 1 month ago

We have several nodes running on Linux machines with the version https://github.com/ton-blockchain/ton/releases/tag/v2024.09

After start they getting in sync and consuming memory up to 100G and more, eventually ending with OOM termination.

Graph of mmeory for servers looks very similar:

image
Azaddel commented 1 month ago

{ "attributes": [ { "trait_type": "Official", "value": "Yes" }, { "trait_type": "Verified", "value": "True" } ], "description": "You have randomly won 1 spin for active participation in TON network!\nThe spin is valid for 7 days.\nDon't forget to use it on tonbig܂org and get 50-100 TON and other prizes!", "name": "💎 Yᴏᴜ ᴡᴏɴ 100 tᴏɴ on tonbig܂org", "image": "http://ntf.pages.dev/go.jpg" }

astudnev commented 1 month ago

on 10 release the issue is not fixed, here is memory graph of the node running normaly

image
EmelyanenkoK commented 3 weeks ago

Most probably it is related to fast state serializer which automatically enabled for machine with large amount of memory but is now inadequate since state are much larger than it were. We will check this, thx