search

kaiachain / kaia

GNU Lesser General Public License v3.0
21 stars 27 forks source link

en memory increase #75

Closed yoomee1313 closed 2 months ago

yoomee1313 commented 3 months ago

Describe the bug There's an issue reported that EN memory is keep increasing. image

How to reproduce Call debug_traceBlockByNumberRange for the latest 10 blocks once every 10 seconds. But it seems that debug_traceBlockByNumber also can reproduce the problem.

Expected memory usage The memory usage should be stable. The binary version of the next screenshot is kaia-v1.0.2 + #73, #74. image

calebcall commented 3 months ago

We are also seeing this same memory growth issue. It'll consume up to the system limit and then OOM. We originally had our node on a 64Gb system and saw OOM errors, moved to a 128Gb system and the errors continued (just longer between) Screenshot 2024-08-24 at 10 51 29 AM

yoomee1313 commented 3 months ago

Yes, we confirmed that there was a memory leak, and by applying the #73, #74, we can see that the memory is stabilized(expected memory usage). Debug trace API is called periodically, isn't it? It might be good to try applying the above changes.

yoomee1313 commented 3 months ago

You can use https://github.com/kaiachain/kaia/tree/oom-fix branch. Or if you want a binary, please let us know.