search

skalenetwork / skaled

Running more than 20 production blockchains, SKALED is Ethereum-compatible, high performance C++ Proof-of-Stake client, tools and libraries. Uses SKALE consensus as a blockchain consensus core. Includes dynamic Oracle. Implements file storage and retrieval as an EVM extension.
https://skale.network
GNU General Public License v3.0
85 stars 40 forks source link

int instead of size_t in HEAP_MEM #1982

Open oleksandrSydorenkoJ opened 2 months ago

oleksandrSydorenkoJ commented 2 months ago

Describe the bug Actual for core and indexer nodes Related to https://github.com/skalenetwork/internal-support/issues/931

Versions skalenetwork/schain:3.19.1

Preconditions: Skale chain Medium type with 16 active nodes Enabled MTM Connected skale-proxy to the Skale chain Connected at least 1 indexer node to the Skale Chain

To Reproduce

  1. Run load tests on 1 chain for 3 days
  2. Wait until the Skaled HEAP_MEM exceeds the 2137605504 bytes
  3. Check skaled logs

Expected behavior Skaled should print the current HEAP_MEM size in the log according to the actual usage.

Actual state: Skaled crashes with OOM when HEAP_MEM exceeds the size of 2137605504.

[2024-09-07 23:01:41.820] [8:main] [info] 1460627:RUSAGE:USER_CPU:82694:896343:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:2136553264:
[2024-09-07 23:02:41.348] [8:main] [info] 1460647:RUSAGE:USER_CPU:82704:47259:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2141510352:
[2024-09-07 23:03:40.929] [8:main] [info] 1460667:RUSAGE:USER_CPU:82712:804585:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:2147011456:
[2024-09-07 23:04:33.644] [8:main] [info] 1460687:RUSAGE:USER_CPU:82721:152020:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2133852704:
[2024-09-07 23:05:35.895] [8:main] [info] 1460707:RUSAGE:USER_CPU:82731:263291:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2143199120:
[2024-09-07 23:06:34.832] [8:main] [info] 1460727:RUSAGE:USER_CPU:82742:42802:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2133314416:
[2024-09-07 23:07:27.557] [8:main] [info] 1460747:RUSAGE:USER_CPU:82749:227899:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2129199312:
[2024-09-07 23:08:24.426] [8:main] [info] 1460767:RUSAGE:USER_CPU:82756:505229:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2129500560:
[2024-09-07 23:09:22.378] [8:main] [info] 1460787:RUSAGE:USER_CPU:82766:534775:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2132925968:
[2024-09-07 23:10:19.730] [8:main] [info] 1460807:RUSAGE:USER_CPU:82777:12142:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2116138928:
[2024-09-07 23:11:18.786] [8:main] [info] 1460827:RUSAGE:USER_CPU:82786:233081:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2127419952:
[2024-09-07 23:12:13.705] [8:main] [info] 1460847:RUSAGE:USER_CPU:82795:301516:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2130717328:
[2024-09-07 23:13:11.449] [8:main] [info] 1460867:RUSAGE:USER_CPU:82804:21806:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2118161120:
[2024-09-07 23:14:07.450] [8:main] [info] 1460887:RUSAGE:USER_CPU:82812:413539:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2123569008:
[2024-09-07 23:15:06.004] [8:main] [info] 1460907:RUSAGE:USER_CPU:82823:79012:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2116206400:
[2024-09-07 23:16:02.783] [8:main] [info] 1460927:RUSAGE:USER_CPU:82831:595735:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2111923632:
[2024-09-07 23:16:58.166] [8:main] [info] 1460947:RUSAGE:USER_CPU:82839:435290:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2100281200:
[2024-09-07 23:17:50.239] [8:main] [info] 1460967:RUSAGE:USER_CPU:82846:12259:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2104603488:
[2024-09-07 23:18:43.550] [8:main] [info] 1460987:RUSAGE:USER_CPU:82853:733131:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2090585952:
[2024-09-07 23:19:41.414] [8:main] [info] 1461007:RUSAGE:USER_CPU:82862:406490:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2090125616:
[2024-09-07 23:20:30.644] [8:main] [info] 1461027:RUSAGE:USER_CPU:82870:81147:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2099123824:
[2024-09-07 23:20:40.989] [8:main] [info] 1461047:RUSAGE:USER_CPU:82877:472221:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2106375392:
[2024-09-07 23:21:00.233] [8:main] [info] 1461067:RUSAGE:USER_CPU:82885:18391:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2091413712:
[2024-09-07 23:21:12.708] [8:main] [info] 1461087:RUSAGE:USER_CPU:82894:72138:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2082410896:
[2024-09-07 23:21:33.132] [8:main] [info] 1461107:RUSAGE:USER_CPU:82903:47809:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2089827616:
[2024-09-07 23:21:43.216] [8:main] [info] 1461127:RUSAGE:USER_CPU:82912:103523:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2073349312:
[2024-09-07 23:21:53.437] [8:main] [info] 1461147:RUSAGE:USER_CPU:82921:597710:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2054117520:
[2024-09-07 23:22:05.944] [8:main] [info] 1461167:RUSAGE:USER_CPU:82931:753048:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2127760960:
[2024-09-07 23:22:15.721] [8:main] [info] 1461187:RUSAGE:USER_CPU:82940:721089:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2134320576:
[2024-09-07 23:22:26.443] [8:main] [info] 1461207:RUSAGE:USER_CPU:82950:655641:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2112446576:
[2024-09-07 23:22:34.788] [8:main] [info] 1461227:RUSAGE:USER_CPU:82958:128740:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2102600704:
[2024-09-07 23:22:47.830] [8:main] [info] 1461247:RUSAGE:USER_CPU:82968:846830:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2140733296:
[2024-09-07 23:23:13.829] [8:main] [info] 1461267:RUSAGE:USER_CPU:82976:415155:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:2134827056:
[2024-09-07 23:23:59.178] [8:main] [info] 1461287:RUSAGE:USER_CPU:82985:606926:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2146329744:
[2024-09-07 23:24:35.791] [8:main] [info] 1461307:RUSAGE:USER_CPU:82994:859963:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2139914560:
[2024-09-07 23:25:24.913] [8:main] [info] 1461327:RUSAGE:USER_CPU:83006:419935:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:-2147226704:
[2024-09-07 23:25:56.387] [8:main] [info] 1461347:RUSAGE:USER_CPU:83015:405461:PROCESS_RSS:3465996:SWAPPED_MEM:0:HEAP_MEM:2137478368:
dimalit commented 1 month ago

It's already int and that is the problem: sizeof(int)==4. And it comes from struct mallinfo