search

MystenLabs / sui

Sui, a next-generation smart contract platform with high throughput, low latency, and an asset-oriented programming model powered by the Move programming language
https://sui.io
Apache License 2.0
5.97k stars 11.09k forks source link

Sui oom-kill #18768

Open zhy827827 opened 1 month ago

zhy827827 commented 1 month ago

Sui Node version: 1.28.4-fc0623927416 hardware
CPU 12C MEM 64G Disk 4T

Start the Sui node, the process will be killed after 2 minutes, and then restart

Logs:

ul 23 07:37:07 sui-mainnet sui-node[11139]: 2024-07-23T07:37:07.897686Z  WARN request{route=/sui.StateSync/GetCheckpointSummary remote_peer_id=6f0b2508 direction=outbound}: anemo_tower::trace::on_failure: response failed error=Error: connection lost latency=247 ms
Jul 23 07:37:07 sui-mainnet sui-node[11139]: 2024-07-23T07:37:07.897697Z  WARN request{route=/sui.StateSync/GetCheckpointSummary remote_peer_id=6f0b2508 direction=outbound}: anemo_tower::trace::on_failure: response failed error=Error: connection lost latency=248 ms
Jul 23 07:37:07 sui-mainnet sui-node[11139]: 2024-07-23T07:37:07.999369Z  WARN sui_core::checkpoints::checkpoint_executor: Received no new synced checkpoints for 5s. Next checkpoint to be scheduled: 40069882
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683501] sui-node-runtim invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683505] CPU: 9 PID: 11149 Comm: sui-node-runtim Not tainted 5.15.0-116-generic #126-Ubuntu
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683507] Hardware name: GIGABYTE MX33-BS1-V1/MX33-BS1-V1, BIOS F09d 08/27/2023
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683508] Call Trace:
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683509]  <TASK>
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683510]  show_stack+0x52/0x5c
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683513]  dump_stack_lvl+0x4a/0x63
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683515]  dump_stack+0x10/0x16
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683516]  dump_header+0x53/0x228
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683518]  oom_kill_process.cold+0xb/0x10
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683520]  out_of_memory+0x106/0x2e0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683522]  __alloc_pages_slowpath.constprop.0+0x9a0/0xac0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683524]  __alloc_pages+0x311/0x330
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683525]  alloc_pages+0x9e/0x1e0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683527]  __page_cache_alloc+0x7e/0x90
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683528]  pagecache_get_page+0x152/0x590
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683529]  ? page_cache_ra_unbounded+0x163/0x210
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683531]  filemap_fault+0x488/0xab0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683532]  ? filemap_map_pages+0x309/0x400
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683533]  __do_fault+0x39/0x120
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683534]  do_read_fault+0xeb/0x160
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683535]  do_fault+0xa0/0x2e0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683536]  handle_pte_fault+0x1cd/0x240
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683537]  __handle_mm_fault+0x405/0x6f0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683539]  handle_mm_fault+0xd8/0x2c0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683540]  do_user_addr_fault+0x1c9/0x670
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683542]  exc_page_fault+0x77/0x170
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683543]  asm_exc_page_fault+0x27/0x30
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683545] RIP: 0033:0x5589c42b3d41
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683549] Code: Unable to access opcode bytes at RIP 0x5589c42b3d17.
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683549] RSP: 002b:00007fb0cf04d240 EFLAGS: 00010202
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683550] RAX: 0000000000000000 RBX: 00007fb09c003c40 RCX: 0000000000000010
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683551] RDX: 0000000000000002 RSI: 00007fb0cf04d2c8 RDI: 00007fb09c003da8
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683552] RBP: 00007fb0cf04d290 R08: 45df4669f047c67f R09: 0000000000000000
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683552] R10: 00007f9d7ac8d570 R11: 23b052db12ba7095 R12: 00007faeda58fe40
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683553] R13: 00007fb0cf04d500 R14: 00007fb0cf04d3c0 R15: 00007fb0cf04d370
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683554]  </TASK>
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683555] Mem-Info:
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556] active_anon:326 inactive_anon:16233831 isolated_anon:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  active_file:464 inactive_file:323 isolated_file:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  unevictable:6936 dirty:0 writeback:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  slab_reclaimable:31407 slab_unreclaimable:19142
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  mapped:2264 shmem:360 pagetables:33116 bounce:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  kernel_misc_reclaimable:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683556]  free:81482 free_pcp:0 free_cma:0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683558] Node 0 active_anon:1304kB inactive_anon:64935324kB active_file:1856kB inactive_file:1292kB unevictable:27744kB isolated(anon):0kB isolated(file):0kB mapped:9056kB dirty:0kB writeback:0kB shmem:1440kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:5792kB pagetables:132464kB all_unreclaimable? yes
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683560] Node 0 DMA free:11264kB min:12kB low:24kB high:36kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683563] lowmem_reserve[]: 0 2261 64057 64057 64057
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683565] Node 0 DMA32 free:249524kB min:2384kB low:4696kB high:7008kB reserved_highatomic:0KB active_anon:0kB inactive_anon:2163504kB active_file:0kB inactive_file:48kB unevictable:0kB writepending:0kB present:2490048kB managed:2423956kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683567] lowmem_reserve[]: 0 0 61795 61795 61795
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683569] Node 0 Normal free:65140kB min:65180kB low:128456kB high:191732kB reserved_highatomic:0KB active_anon:1304kB inactive_anon:62771820kB active_file:1360kB inactive_file:1408kB unevictable:27744kB writepending:0kB present:64487424kB managed:63287848kB mlocked:27744kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683571] lowmem_reserve[]: 0 0 0 0 0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683573] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683577] Node 0 DMA32: 68*4kB (UME) 48*8kB (UME) 75*16kB (UME) 57*32kB (UME) 31*64kB (UME) 10*128kB (UME) 8*256kB (UME) 4*512kB (UME) 1*1024kB (M) 0*2048kB 58*4096kB (UM) = 249632kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683584] Node 0 Normal: 924*4kB (UME) 814*8kB (UME) 585*16kB (UME) 453*32kB (UME) 250*64kB (UME) 122*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 65680kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683589] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683590] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683591] 3385 total pagecache pages
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683592] 0 pages in swap cache
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683592] Swap cache stats: add 0, delete 0, find 0/0
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683593] Free swap  = 0kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683593] Total swap = 0kB
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683593] 16748366 pages RAM
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683594] 0 pages HighMem/MovableOnly
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683594] 316575 pages reserved
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683594] 0 pages hwpoisoned
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683595] Tasks state (memory values in pages):
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683595] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683600] [    544]     0   544    58867     1087   466944        0          -250 systemd-journal
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683602] [    580]     0   580    72360     6806   110592        0         -1000 multipathd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683603] [    594]     0   594     2894      870    61440        0         -1000 systemd-udevd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683604] [    734]     0   734      982      383    45056        0             0 mdadm
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683606] [    777]   103   777    22341      214    69632        0             0 systemd-timesyn
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683607] [    815]   100   815     4065      654    65536        0             0 systemd-network
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683608] [    817]   101   817     6418     1725    90112        0             0 systemd-resolve
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683608] [    842]     0   842     1822      625    49152        0             0 cron
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683609] [    843]   102   843     2173      639    57344        0          -900 dbus-daemon
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683610] [    847]     0   847   101953     7166   167936        0             0 glances
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683611] [    851]     0   851    20711      699    57344        0             0 irqbalance
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683612] [    855]     0   855     8271     2880    98304        0             0 networkd-dispat
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683613] [    856]   104   856    55601      613    81920        0             0 rsyslogd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683614] [    895]     0   895   514961     2884   348160        0          -900 snapd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683615] [    950]     0   950     5931      489    69632        0             0 systemd-logind
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683616] [    972]     0   972    31600      663    90112        0             0 thermald
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683617] [    985]     0   985     3859     1179    69632        0         -1000 sshd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683618] [   1014]     0  1014    27527     2798   114688        0             0 unattended-upgr
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683619] [   1027]     0  1027     1555      191    45056        0             0 agetty
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683620] [   1031]     0  1031     1544      208    49152        0             0 agetty
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683621] [   8881]     0  8881     4231     1267    73728        0             0 sshd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683622] [   8884]  1000  8884     4300     1085    77824        0             0 systemd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683623] [   8885]  1000  8885    42652     1272    98304        0             0 (sd-pam)
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683624] [   8945]  1000  8945     4493     1268    77824        0             0 sshd
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683625] [   8946]  1000  8946     2407     1103    61440        0             0 bash
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683626] [  11139]  1000 11139 20573315 16210334 133160960        0             0 sui-node
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683627] [  11779]  1000 11779     1556      173    45056        0             0 tail
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683628] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=sui.service,mems_allowed=0,global_oom,task_memcg=/system.slice/sui.service,task=sui-node,pid=11139,uid=1000
Jul 23 07:37:14 sui-mainnet kernel: [ 2764.683741] Out of memory: Killed process 11139 (sui-node) total-vm:82293260kB, anon-rss:64841336kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:130040kB oom_score_adj:0
Jul 23 07:37:15 sui-mainnet systemd[1]: sui.service: Main process exited, code=killed, status=9/KILL
Jul 23 07:37:15 sui-mainnet systemd[1]: sui.service: Failed with result 'signal'.
Jul 23 07:37:15 sui-mainnet systemd[1]: sui.service: Consumed 2min 35.074s CPU time.
Jul 23 07:37:45 sui-mainnet systemd[1]: sui.service: Scheduled restart job, restart counter is at 3.
Jul 23 07:37:45 sui-mainnet systemd[1]: Stopped sui.
Jul 23 07:37:45 sui-mainnet systemd[1]: sui.service: Consumed 2min 35.074s CPU time.
Jul 23 07:37:45 sui-mainnet systemd[1]: Started sui.
Jul 23 07:37:45 sui-mainnet sui-node[11823]: 2024-07-23T07:37:45.947990Z  INFO sui_node: Sui Node version: 1.28.4-fc0623927416
Jul 23 07:37:45 sui-mainnet sui-node[11823]: 2024-07-23T07:37:45.948379Z  INFO sui_node: Supported protocol versions: Some(SupportedProtocolVersions { min: ProtocolVersion(1), max: ProtocolVersion(51) })
Jul 23 07:37:45 sui-mainnet sui-node[11823]: 2024-07-23T07:37:45.948385Z  INFO sui_node: Started Prometheus HTTP endpoint at 127.0.0.1:9184
Jul 23 07:37:45 sui-mainnet sui-node[11823]: 2024-07-23T07:37:45.949354Z  INFO sui_node: Initializing sui-node listening on /dns/0.0.0.0/tcp/8080/http node=k#9063e21b9b938894bd699d0e275ae7c6971b4f484fe79eeb645f649dadfe7cce99bd928b6f504530e6448b5732f692ee13e3317b831b1f7cda6c8ef32dcd181f15253cc0f3c3068432fb4c25037d76981d29dcee0a02c49b3f43c9e991e853ee
Jul 23 07:37:48 sui-mainnet sui-node[11823]: 2024-07-23T07:37:48.499198Z  INFO sui_core::authority::authority_store: Loading epoch start config from DB

How can I optimize it?

It's strange that my other two servers are running very well. The software and hardware configurations are the same

sui

mwtian commented 1 month ago

Before the node was killed, is it possible to check if there are large values on any label of these prometheus metrics:

Please let us know total values and the largest labels on these metrics, and we can check if they indicate issues.

zhy827827 commented 1 month ago

sorry,i don't use prometheus metrics.

mwtian commented 1 month ago

It should be straight forward to set grafana up for sui-node. This post is from awhile ago but could still be useful: https://forums.sui.io/t/monitoring-and-alerts-integration-for-your-node/15449

Alternatively, we can try using jemalloc and jeprof to profile the memory. On Ubuntu, you can:

  1. Install jemalloc, jeprof and graphviz: sudo apt install libjemalloc-dev graphviz
  2. Figure out the path to the jemalloc library, usually /usr/lib/x86_64-linux-gnu/libjemalloc.so
  3. Create a directory to dump memory profiles, e.g. /opt/sui/jemalloc/.
  4. Start sui-node with LD_PRELOAD /usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF=prof:true,prof_prefix:/opt/sui/jeprof/jeprof.out,lg_prof_interval:34. This basically dumps a memory profile after some memory allocations.
  5. After sui-node memory usage becomes too high, find the latest profile from ls -1t /opt/sui/jeprof/ | head -20
  6. Visualize the profile with the exact sui-node binary that generated it: sudo jeprof --svg /opt/sui/bin/sui-node /opt/sui/jeprof/<file name chosen from above> > jeprof.svg

Posting the jeprof.svg would help us identify the issue.