arriven
commented
2 years ago How much memory do you have?
Open MidnightRAT opened 2 years ago
arriven
commented
2 years ago How much memory do you have?
deputinizer
commented
2 years ago I have 2 GB of RAM and still getting OOM after a few days...
info attacking {"target": "http://xxie"}
info attacking {"target": "http://xxsa"}
info attacking {"target": "http://xxtil/"}
KilledVersion 0.9.9
arriven
commented
2 years ago A few days will be interesting to debug
deputinizer
commented
2 years ago Next day... next kill... i have to pprof before it gets killed
| --- | --- | --- | --- | --- |
| Total | 13608405 | 13336567 | 4016925 | 724.90 MB |
Killed
ubuntu@vps-123456789:~/stoppropaganda$Latest version 0.9.10
deputinizer
commented
2 years ago Also, I got a very uncomfortable email...
Is there a way to lower the rate of TCP SYN / per second ?
Dear Customer,
Abnormal activity has been detected on your VPS vps1234.ovh.net.
As this constitutes a breach of contract, your virtual server vps1234.ovh.net
has been blocked.
You will find the logs brought up by our system below, which led to this alert.
- START OF ADDITIONAL INFORMATION -
Attack detail : 8Kpps/4Mbps
dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason
2022.06.18 02:30:52 CEST vps1234:55046 84.252.129.113:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:36158 195.128.49.210:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:59934 172.67.145.235:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:48778 195.128.137.3:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:38424 31.31.198.231:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:38466 104.21.89.179:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58508 84.42.111.137:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:54342 31.31.198.225:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:41018 31.31.196.201:443 TCP SYN 56 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:37354 104.21.21.95:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:55052 84.252.129.113:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58170 37.140.192.179:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:59990 172.67.145.235:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:46116 37.140.192.54:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58364 172.67.180.134:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:41332 37.140.192.89:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:42900 185.26.122.77:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:37134 77.222.56.143:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58306 172.67.180.134:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:39082 194.105.131.43:80 TCP SYN 60 ATTACK:TCP_SYN And yet another panic... funny it's on a single-thread machine :laughing: so it's not race condition :worried:
... 6969 more lines ...
goroutine 20488 [sleep]:
time.Sleep(0x14f46b0400)
/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.(*HostClient).connsCleaner(0xc00db6ad00)
/home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/client.go:1711 +0x58
created by github.com/valyala/fasthttp.(*HostClient).acquireConn
/home/runner/go/pkg/mod/github.com/valyala/fasthttp@v1.34.0/client.go:1602 +0x218Next time running ./db1000n 2> errors.txt
arriven
commented
2 years ago Also, I got a very uncomfortable email...
Is there a way to lower the rate of TCP SYN / per second ?
Interesting, there isn't any tcp syn attacks implemented in the app because they are generally blocked by the provider. The only thing that could be treated like this is the amount of attempts to build a connection with the target. You can try using the general rate limiting to work around that but it'll decrease the overall performance of the app (--min-interval
deputinizer
commented
2 years ago amount of attempts to build a connection
well that's called Dial
In stoppropaganda i've implmented rate limiting https://github.com/erkexzcx/stoppropaganda#dialspersecond
Edit: Here was the PR https://github.com/arriven/db1000n/pull/535/commits/025104a97a6f98f61bb4b554ca6437a649b29119#diff-f1c4adfa861d40deef193a963cc05fe1180e22fa8d2c4195c04b40b989fba4f3R238 Yeah, this PR was a little bloated :stuck_out_tongue_winking_eye:
arriven
commented
2 years ago Yeah, there's already a rate limiting in place, I just didn't treat dial as a special case but maybe I should
arriven
commented
2 years ago @deputinizer thinking about it more, I think you can already achieve the same results with existing configuration options. There's exponential backoff present on errors which you can configure to have higher initial timeout (--backoff-timeout, default is 1 microsecond) and adapt backoff-limit/backoff-multiplier accordingly so that it doesn't spin to thousands of seconds
arriven
commented
2 years ago regarding the actual memory issue, I've been debugging todays config for couple of hours and the memory usage was only fluctuating for couple of MBs. Testing it out on my benchmark server revealed that it's possible to increase the memory usage for one of the jobs by increasing the response size from the target. We were already planning to replace that job with a more effective one, I'll update this thread once we do that
deputinizer
commented
2 years ago arriven
commented
2 years ago @MidnightRAT @deputinizer should be fixed now, poke me if you still encounter it
MidnightRAT
commented
2 years ago [ 3449.257924] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
[ 3449.257933] CPU: 0 PID: 405 Comm: systemd-journal Not tainted 5.13.0-1036-oracle #43~20.04.1-Ubuntu
[ 3449.257936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[ 3449.257938] Call Trace:
[ 3449.257940] <TASK>
[ 3449.257943] dump_stack+0x7d/0x9c
[ 3449.257949] dump_header+0x4f/0x1f6
[ 3449.257953] oom_kill_process.cold+0xb/0x10
[ 3449.257957] out_of_memory+0x1cf/0x520
[ 3449.257961] __alloc_pages_slowpath.constprop.0+0xc86/0xd50
[ 3449.257967] __alloc_pages+0x2ce/0x320
[ 3449.257970] alloc_pages+0x87/0xe0
[ 3449.257973] __page_cache_alloc+0x89/0xb0
[ 3449.257977] pagecache_get_page+0x218/0x4d0
[ 3449.257980] filemap_fault+0x5bf/0x9b0
[ 3449.257981] ? do_set_pte+0xc8/0x140
[ 3449.257984] ? next_uptodate_page+0x160/0x270
[ 3449.257987] ext4_filemap_fault+0x32/0x50
[ 3449.257991] __do_fault+0x3e/0xc0
[ 3449.257994] do_fault+0x1e6/0x400
[ 3449.257996] __handle_mm_fault+0x619/0x8e0
[ 3449.257999] handle_mm_fault+0xda/0x2b0
[ 3449.258001] do_user_addr_fault+0x1bb/0x650
[ 3449.258005] exc_page_fault+0x7d/0x170
[ 3449.258008] ? asm_exc_page_fault+0x8/0x30
[ 3449.258012] asm_exc_page_fault+0x1e/0x30
[ 3449.258015] RIP: 0033:0x7f1e00753f7b
[ 3449.258022] Code: Unable to access opcode bytes at RIP 0x7f1e00753f51.
[ 3449.258024] RSP: 002b:00007ffc2419f118 EFLAGS: 00010246
[ 3449.258027] RAX: 0000000000000000 RBX: 0000000000000578 RCX: 0000000000000006
[ 3449.258029] RDX: 0000000000000006 RSI: 00007f1e008b11bc RDI: 00007f1dff64f5b8
[ 3449.258030] RBP: 000055eb7e86d260 R08: 00000000000000e8 R09: 000055eb7e83df40
[ 3449.258032] R10: 00007f1dff6442e0 R11: 00007f1dfc104db0 R12: 000000000039d5b8
[ 3449.258033] R13: 00007ffc2419f148 R14: 00000000020dad70 R15: 00007f1dff64f5b8
[ 3449.258036] </TASK>
[ 3449.258036] Mem-Info:
[ 3449.258038] active_anon:251 inactive_anon:173988 isolated_anon:0
active_file:0 inactive_file:1883 isolated_file:32
unevictable:5292 dirty:0 writeback:0
slab_reclaimable:6046 slab_unreclaimable:16938
mapped:2379 shmem:271 pagetables:1079 bounce:0
free:14684 free_pcp:63 free_cma:0
[ 3449.258043] Node 0 active_anon:1004kB inactive_anon:695952kB active_file:0kB inactive_file:7532kB unevictable:21168kB isolated(anon):0kB isolated(file):0kB mapped:9516kB dirty:0kB writeback:0kB shmem:1084kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 278528kB writeback_tmp:0kB kernel_stack:3104kB pagetables:4316kB all_unreclaimable? yes
[ 3449.258048] Node 0 DMA free:4308kB min:680kB low:848kB high:1016kB reserved_highatomic:0KB active_anon:0kB inactive_anon:9552kB active_file:0kB inactive_file:180kB unevictable:0kB writepending:0kB present:15004kB managed:14372kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 3449.258053] lowmem_reserve[]: 0 913 913 913 913
[ 3449.258057] Node 0 DMA32 free:54428kB min:44372kB low:55464kB high:66556kB reserved_highatomic:10240KB active_anon:1004kB inactive_anon:686400kB active_file:164kB inactive_file:7036kB unevictable:21168kB writepending:0kB present:1027952kB managed:974964kB mlocked:21168kB bounce:0kB free_pcp:252kB local_pcp:248kB free_cma:0kB
[ 3449.258063] lowmem_reserve[]: 0 0 0 0 0
[ 3449.258066] Node 0 DMA: 4*4kB (U) 4*8kB (U) 5*16kB (UE) 3*32kB (UE) 6*64kB (UME) 3*128kB (UME) 3*256kB (UME) 1*512kB (E) 2*1024kB (UE) 0*2048kB 0*4096kB = 4320kB
[ 3449.258081] Node 0 DMA32: 2440*4kB (UME) 1846*8kB (UME) 590*16kB (UME) 173*32kB (ME) 85*64kB (ME) 38*128kB (ME) 20*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54928kB
[ 3449.258095] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 3449.258097] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 3449.258098] 4551 total pagecache pages
[ 3449.258099] 0 pages in swap cache
[ 3449.258100] Swap cache stats: add 0, delete 0, find 0/0
[ 3449.258101] Free swap = 0kB
[ 3449.258102] Total swap = 0kB
[ 3449.258102] 260739 pages RAM
[ 3449.258103] 0 pages HighMem/MovableOnly
[ 3449.258104] 13405 pages reserved
[ 3449.258104] 0 pages hwpoisoned
[ 3449.258105] Tasks state (memory values in pages):
[ 3449.258106] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 3449.258108] [ 405] 0 405 28618 1092 225280 0 -250 systemd-journal
[ 3449.258112] [ 428] 0 428 4943 783 61440 0 -1000 systemd-udevd
[ 3449.258116] [ 566] 0 566 70051 4500 86016 0 -1000 multipathd
[ 3449.258119] [ 657] 0 657 624 128 45056 0 0 bpfilter_umh
[ 3449.258122] [ 659] 112 659 1777 720 53248 0 0 rpcbind
[ 3449.258125] [ 660] 102 660 22493 687 77824 0 0 systemd-timesyn
[ 3449.258128] [ 714] 100 714 6645 677 73728 0 0 systemd-network
[ 3449.258152] [ 716] 101 716 5965 1681 90112 0 0 systemd-resolve
[ 3449.258154] [ 762] 0 762 60325 970 102400 0 0 accounts-daemon
[ 3449.258157] [ 763] 103 763 1951 805 53248 0 -900 dbus-daemon
[ 3449.258160] [ 769] 0 769 20458 605 65536 0 0 irqbalance
[ 3449.258162] [ 771] 0 771 7404 2848 90112 0 0 networkd-dispat
[ 3449.258164] [ 773] 0 773 59107 635 90112 0 0 polkitd
[ 3449.258166] [ 774] 104 774 56088 364 77824 0 0 rsyslogd
[ 3449.258169] [ 778] 584788 778 178846 1559 114688 0 0 updater
[ 3449.258171] [ 780] 584788 780 178881 1591 118784 0 0 agent
[ 3449.258174] [ 783] 0 783 4155 887 69632 0 0 systemd-logind
[ 3449.258176] [ 784] 0 784 98718 1360 139264 0 0 udisksd
[ 3449.258178] [ 799] 0 799 1279 54 45056 0 0 iscsid
[ 3449.258181] [ 800] 0 800 1405 1365 49152 0 -17 iscsid
[ 3449.258183] [ 816] 0 816 2136 579 49152 0 0 cron
[ 3449.258185] [ 822] 0 822 3045 709 69632 0 -1000 sshd
[ 3449.258188] [ 824] 0 824 950 519 40960 0 0 atd
[ 3449.258190] [ 842] 0 842 1840 424 45056 0 0 agetty
[ 3449.258192] [ 848] 0 848 1459 386 53248 0 0 agetty
[ 3449.258194] [ 865] 0 865 27028 2777 110592 0 0 unattended-upgr
[ 3449.258197] [ 868] 0 868 78615 1103 110592 0 0 ModemManager
[ 3449.258199] [ 1068] 584788 1068 179221 2099 118784 0 0 gomon
[ 3449.258201] [ 1079] 0 1079 3451 1019 65536 0 0 sshd
[ 3449.258204] [ 1082] 1001 1082 4592 1190 77824 0 0 systemd
[ 3449.258206] [ 1083] 1001 1083 25832 875 98304 0 0 (sd-pam)
[ 3449.258208] [ 1179] 1001 1179 3485 368 65536 0 0 sshd
[ 3449.258211] [ 1180] 1001 1180 2515 899 53248 0 0 bash
[ 3449.258213] [ 1192] 1001 1192 2515 575 49152 0 0 bash
[ 3449.258215] [ 1199] 1001 1199 387627 152709 1671168 0 0 db1000n
[ 3449.258217] [ 1240] 1001 1240 177110 1254 143360 0 0 launcher-disbal
[ 3449.258220] [ 1295] 1001 1295 177640 1683 114688 0 0 app
[ 3449.258222] [ 1534] 0 1534 8513 1082 61440 0 -900 snapd
[ 3449.258225] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/session-1.scope,task=db1000n,pid=1199,uid=1001
[ 3449.258263] Out of memory: Killed process 1199 (db1000n) total-vm:1550508kB, anon-rss:610836kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:1632kB oom_score_adj:0
[ 3449.359951] oom_reaper: reaped process 1199 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 3455.762261] loop10: detected capacity change from 0 to 8
root@instance-20220226-1719:~# free
total used free shared buff/cache available
Mem: 989336 190572 491268 1092 307496 655840
Swap: 0 0 0
heximcz
commented
2 years ago [ 1937.576964] db1000n invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[ 1937.576996] CPU: 2 PID: 850 Comm: db1000n Tainted: G C 5.10.103-v7l+ #1529
[ 1937.577000] Hardware name: BCM2711
[ 1937.577006] Backtrace:
[ 1937.577041] [<c0b875f0>] (dump_backtrace) from [<c0b87984>] (show_stack+0x20/0x24)
[ 1937.577050] r7:ffffffff r6:00000000 r5:60000013 r4:c12e6fd8
[ 1937.577059] [<c0b87964>] (show_stack) from [<c0b8bd74>] (dump_stack+0xcc/0xf8)
[ 1937.577067] [<c0b8bca8>] (dump_stack) from [<c0b8a004>] (dump_header+0x64/0x208)
[ 1937.577073] r10:c12051c8 r9:00000cc0 r8:00000000 r7:c0e3c824 r6:c40b6d00 r5:c1ac1f00
[ 1937.577078] r4:cb9e9cd0 r3:6e4df213
[ 1937.577097] [<c0b89fa0>] (dump_header) from [<c03ba5f8>] (oom_kill_process+0x1b4/0x1c0)
[ 1937.577102] r7:c0e3c824 r6:cb9e9cd0 r5:c1ac2480 r4:c1ac1f00
[ 1937.577112] [<c03ba444>] (oom_kill_process) from [<c03bb170>] (out_of_memory+0x2b8/0x390)
[ 1937.577117] r7:c1208600 r6:c1205048 r5:c1ac1f00 r4:cb9e9cd0
[ 1937.577127] [<c03baeb8>] (out_of_memory) from [<c0410030>] (__alloc_pages_nodemask+0x7ec/0x1184)
[ 1937.577133] r7:c1338550 r6:00001000 r5:0001494e r4:00000000
[ 1937.577147] [<c040f844>] (__alloc_pages_nodemask) from [<c09fc540>] (skb_page_frag_refill+0xe4/0x10c)
[ 1937.577154] r10:7aa42cec r9:0000006c r8:d2925b00 r7:cb9e9e60 r6:00000040 r5:00000cc0
[ 1937.577158] r4:c3e863d0
[ 1937.577165] [<c09fc45c>] (skb_page_frag_refill) from [<c09fcdb4>] (sk_page_frag_refill+0x24/0x78)
[ 1937.577169] r5:d2925c04 r4:d2925b00
[ 1937.577181] [<c09fcd90>] (sk_page_frag_refill) from [<c0ab2ae4>] (tcp_sendmsg_locked+0x4c0/0xd98)
[ 1937.577186] r5:d2925c04 r4:c869e000
[ 1937.577195] [<c0ab2624>] (tcp_sendmsg_locked) from [<c0ab33f4>] (tcp_sendmsg+0x38/0x4c)
[ 1937.577201] r10:c4484500 r9:c4342540 r8:cb9e9ef8 r7:00000000 r6:000000a9 r5:cb9e9e60
[ 1937.577205] r4:d2925b00
[ 1937.577214] [<c0ab33bc>] (tcp_sendmsg) from [<c0aeba24>] (inet_sendmsg+0x48/0x54)
[ 1937.577219] r7:00000000 r6:000000a9 r5:cb9e9e60 r4:d2925b00
[ 1937.577228] [<c0aeb9dc>] (inet_sendmsg) from [<c09f8390>] (sock_sendmsg+0x44/0x54)
[ 1937.577234] r7:00000000 r6:00000000 r5:c4484500 r4:cb9e9e60
[ 1937.577241] [<c09f834c>] (sock_sendmsg) from [<c09f8444>] (sock_write_iter+0xa4/0xf8)
[ 1937.577245] r5:c1205048 r4:cb9e9ee0
[ 1937.577255] [<c09f83a0>] (sock_write_iter) from [<c0440184>] (vfs_write+0x3ec/0x464)
[ 1937.577262] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:c1205048 r5:000000a9
[ 1937.577266] r4:c4342540
[ 1937.577272] [<c043fd98>] (vfs_write) from [<c04403c4>] (ksys_write+0xcc/0xe0)
[ 1937.577278] r10:000000a9 r9:04f1c4d0 r8:c4342540 r7:00000004 r6:03aecf48 r5:c4342541
[ 1937.577282] r4:c1205048
[ 1937.577289] [<c04402f8>] (ksys_write) from [<c04403f0>] (sys_write+0x18/0x1c)
[ 1937.577294] r10:00000004 r9:cb9e8000 r8:c0200204 r7:00000004 r6:03aecf48 r5:00000000
[ 1937.577298] r4:00000000
[ 1937.577308] [<c04403d8>] (sys_write) from [<c02001e4>] (__sys_trace_return+0x0/0x1c)
[ 1937.577313] Exception stack(0xcb9e9fa8 to 0xcb9e9ff0)
[ 1937.577319] 9fa0: 00000000 00000000 000001af 04f1c4d0 000000a9 00000000
[ 1937.577326] 9fc0: 00000000 00000000 03aecf48 00000004 00000005 00000020 03aecf00 00ecaf68
[ 1937.577332] 9fe0: 04f1c4d3 0adc6be4 000129c0 0009d3ac
[ 1937.577339] Mem-Info:
[ 1937.577355] active_anon:96 inactive_anon:80587 isolated_anon:0
active_file:11029 inactive_file:30902 isolated_file:0
unevictable:4 dirty:6 writeback:0
slab_reclaimable:3288 slab_unreclaimable:10544
mapped:30547 shmem:2169 pagetables:735 bounce:0
free:775436 free_pcp:15 free_cma:80385
[ 1937.577370] Node 0 active_anon:384kB inactive_anon:322348kB active_file:44116kB inactive_file:123608kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:122188kB dirty:24kB writeback:0kB shmem:8676kB writeback_tmp:0kB kernel_stack:1464kB all_unreclaimable? no
[ 1937.577382] DMA free:336620kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:188kB inactive_file:224kB unevictable:0kB writepending:0kB present:786432kB managed:666356kB mlocked:0kB pagetables:248kB bounce:0kB free_pcp:60kB local_pcp:44kB free_cma:321540kB
[ 1937.577390] lowmem_reserve[]: 0 0 3188 3188
[ 1937.577428] HighMem free:2765124kB min:512kB low:20576kB high:40640kB reserved_highatomic:0KB active_anon:384kB inactive_anon:322348kB active_file:43928kB inactive_file:123572kB unevictable:16kB writepending:24kB present:3264512kB managed:3264512kB mlocked:16kB pagetables:2692kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1937.577434] lowmem_reserve[]: 0 0 0 0
[ 1937.577467] DMA: 441*4kB (UMEC) 319*8kB (UMEC) 222*16kB (UMEC) 25*32kB (UMEC) 12*64kB (MEC) 4*128kB (UMEC) 4*256kB (MEC) 5*512kB (MC) 4*1024kB (UMC) 0*2048kB 78*4096kB (C) = 337116kB
[ 1937.577596] HighMem: 1339*4kB (UM) 743*8kB (UM) 368*16kB (UM) 183*32kB (UM) 115*64kB (UM) 39*128kB (UM) 21*256kB (UM) 119*512kB (UM) 71*1024kB (UM) 13*2048kB (UM) 626*4096kB (M) = 2765124kB
[ 1937.577729] 44110 total pagecache pages
[ 1937.577741] 0 pages in swap cache
[ 1937.577749] Swap cache stats: add 0, delete 0, find 0/0
[ 1937.577755] Free swap = 102396kB
[ 1937.577761] Total swap = 102396kB
[ 1937.577767] 1012736 pages RAM
[ 1937.577773] 816128 pages HighMem/MovableOnly
[ 1937.577779] 30019 pages reserved
[ 1937.577785] 81920 pages cma reserved
[ 1937.577792] Tasks state (memory values in pages):
[ 1937.577799] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 1937.577830] [ 137] 0 137 5302 1728 135168 0 0 systemd-journal
[ 1937.577842] [ 165] 0 165 4499 957 40960 0 -1000 systemd-udevd
[ 1937.577863] [ 337] 100 337 5596 707 57344 0 0 systemd-timesyn
[ 1937.577874] [ 399] 0 399 1987 557 40960 0 0 cron
[ 1937.577882] [ 403] 108 403 1442 644 32768 0 0 avahi-daemon
[ 1937.577891] [ 406] 0 406 3253 1439 49152 0 0 systemd-logind
[ 1937.577900] [ 409] 0 409 923 188 36864 0 0 alsactl
[ 1937.577911] [ 416] 104 416 1639 751 40960 0 -900 dbus-daemon
[ 1937.577920] [ 421] 108 421 1442 63 32768 0 0 avahi-daemon
[ 1937.577928] [ 424] 0 424 2678 945 45056 0 0 wpa_supplicant
[ 1937.577936] [ 425] 65534 425 1080 537 36864 0 0 thd
[ 1937.577946] [ 427] 0 427 6914 20 40960 0 0 rngd
[ 1937.577955] [ 428] 0 428 6376 700 49152 0 0 rsyslogd
[ 1937.577965] [ 563] 0 563 228984 9177 180224 0 -999 containerd
[ 1937.577974] [ 564] 0 564 1077 324 32768 0 0 agetty
[ 1937.577985] [ 573] 0 573 2671 1347 45056 0 -1000 sshd
[ 1937.577995] [ 582] 0 582 233188 13976 221184 0 -500 dockerd
[ 1937.578003] [ 621] 0 621 535 32 28672 0 0 hciattach
[ 1937.578012] [ 635] 0 635 2381 851 40960 0 0 bluetoothd
[ 1937.578023] [ 707] 0 707 3134 1603 45056 0 0 sshd
[ 1937.578034] [ 710] 0 710 3642 1750 49152 0 0 systemd
[ 1937.578043] [ 711] 0 711 8807 881 65536 0 0 (sd-pam)
[ 1937.578052] [ 727] 0 727 2090 912 40960 0 0 bash
[ 1937.578063] [ 740] 0 740 2030 665 36864 0 0 tmux: client
[ 1937.578072] [ 742] 0 742 2184 760 36864 0 0 tmux: server
[ 1937.578080] [ 743] 0 743 2093 865 40960 0 0 bash
[ 1937.578087] [ 756] 0 756 229885 8985 172032 0 0 docker
[ 1937.578097] [ 787] 0 787 200854 2137 57344 0 -998 containerd-shim
[ 1937.578105] [ 807] 0 807 270611 65936 1204224 0 0 db1000n
[ 1937.578115] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=f95378fa8d03d2d2754e3f85c98f8d2beb7f61d5af410b7f358ed7a08904fc66,mems_allowed=0,global_oom,task_memcg=/,task=db1000n,pid=807,uid=0
[ 1937.578194] Out of memory: Killed process 807 (db1000n) total-vm:1082444kB, anon-rss:251128kB, file-rss:12616kB, shmem-rss:0kB, UID:0 pgtables:1176kB oom_score_adj:0
[ 1938.117713] oom_reaper: reaped process 807 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================-==============================-============-========================================================
un docker <none> <none> (no description available)
ii docker-ce 5:20.10.17~3-0~raspbian-buster armhf Docker: the open-source application container engine
ii docker-ce-cli 5:20.10.17~3-0~raspbian-buster armhf Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:20.10.17~3-0~raspbian-buster armhf Rootless support for Docker.
ii docker-compose-plugin 2.6.0~raspbian-buster armhf Docker Compose (V2) plugin for the Docker CLI.
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-======================================
un containerd <none> <none> (no description available)
ii containerd.io 1.6.6-1 armhf An open and reliable container runtime
total used free shared buff/cache available
Mem: 3.7Gi 85Mi 3.5Gi 8.0Mi 196Mi 3.5Gi
Swap: 99Mi 0B 99Mi
AlexUkr73
commented
2 years ago the same issue on Ubuntu VM with 2 GB RAM with latest and older db1000n builds:
Jul 16 07:13:11 azurea3 kernel: [64679.724233] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,gl obal_oom,task_memcg=/user.slice,task=db1000n,pid=10373,uid=1000 Jul 16 07:13:11 azurea3 kernel: [64679.724243] Out of memory: Killed process 10373 (db1000n) total-vm:1549092kB, anon-rss:374 792kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1060kB oom_score_adj:0
Could you advice, please? - what syntax and value should be passed to option
-min-interval
regardless of this option's value /absent value - only help displays
Please advice or pont me any how-to with any other ways to reduce memory usage by utility in Ubuntu (e.g. with options examples, etc)
UPD:
this does not help
prlimit --pid <PID_ID> --as=<MEMORY_VALUE>
e.g. prlimit --pid 10632 --as=1024000000
++++++++++++++++ UPD & UPD: script below seems solved the problem by monitoring and re-running 'db1000n' process
until $HOME/./db1000n; do echo "'db1000n' is crashed with exit code $?. Respawning.." >&2 sleep 1 done
arriven
commented
2 years ago I've spent couple of days trying to repro this on my machine but I never got it to consume more than 500 mb of RAM. Did someone monitor the memory usage of db1000n before the oom kill? I'm wondering whether it's a leak that accumulates somewhere or some kind of very weird spike in memory usage, knowing that would help to diagnose the problem
@AlexUkr73 as for -min-interval option something like -min-interval 5s should work
AlexUkr73
commented
2 years ago @arriven Regarding reproducing - I've noticed it is reproduced on VMs with only 1 CPU available and 1 GB RAM (Amazon AWS)
-min-interval 5s- tried everything but didn't expect adding letter after digit ;o)
It will be the good idea to add this statement/example to application's help
MidnightRAT
commented
2 years ago [11514.036137] db1000n invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[11514.036146] CPU: 1 PID: 2173 Comm: db1000n Not tainted 5.15.0-1013-oracle #17~20.04.1-Ubuntu
[11514.036150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[11514.036151] Call Trace:
[11514.036154] <TASK>
[11514.036156] dump_stack_lvl+0x4a/0x5f
[11514.036163] dump_stack+0x10/0x12
[11514.036165] dump_header+0x53/0x221
[11514.036169] oom_kill_process.cold+0xb/0x10
[11514.036172] out_of_memory+0x1dc/0x530
[11514.036176] __alloc_pages_slowpath.constprop.0+0xcd0/0xde0
[11514.036180] ? __alloc_pages_slowpath.constprop.0+0xd84/0xde0
[11514.036184] __alloc_pages+0x2a3/0x2f0
[11514.036187] alloc_pages+0x90/0x110
[11514.036189] __page_cache_alloc+0x83/0xb0
[11514.036192] pagecache_get_page+0x150/0x510
[11514.036195] ? page_cache_ra_unbounded+0x16a/0x210
[11514.036198] filemap_fault+0x523/0xb60
[11514.036201] ? filemap_map_pages+0x134/0x640
[11514.036205] __do_fault+0x40/0x110
[11514.036208] do_fault+0x1f5/0x410
[11514.036211] __handle_mm_fault+0x635/0x850
[11514.036214] handle_mm_fault+0xd8/0x2b0
[11514.036218] do_user_addr_fault+0x1b9/0x650
[11514.036221] exc_page_fault+0x77/0x160
[11514.036225] ? asm_exc_page_fault+0x8/0x30
[11514.036228] asm_exc_page_fault+0x1e/0x30
[11514.036231] RIP: 0033:0x455cf7
[11514.036238] Code: Unable to access opcode bytes at RIP 0x455ccd.
[11514.036239] RSP: 002b:000000c00000b8e8 EFLAGS: 00010216
[11514.036242] RAX: 000000000006a7fc RBX: fffffffffffffffa RCX: 00000000012e8380
[11514.036245] RDX: 00000000000004ea RSI: 00000000000046f4 RDI: 0000000001134a60
[11514.036246] RBP: 000000c00000b8f8 R08: 00000000000004eb R09: 00000000011371b8
[11514.036248] R10: 00007fffa8f3e090 R11: 00000000001f77da R12: 000000c00000b9a0
[11514.036249] R13: ffffffffffffffff R14: 000000c000002000 R15: 0000000000000000
[11514.036252] </TASK>
[11514.036253] Mem-Info:
[11514.036254] active_anon:293 inactive_anon:165819 isolated_anon:0
active_file:9 inactive_file:447 isolated_file:32
unevictable:1396 dirty:0 writeback:0
slab_reclaimable:5947 slab_unreclaimable:17616
mapped:1087 shmem:259 pagetables:877 bounce:0
kernel_misc_reclaimable:0
free:12584 free_pcp:314 free_cma:0
[11514.036260] Node 0 active_anon:1172kB inactive_anon:663276kB active_file:36kB inactive_file:1788kB unevictable:5584kB isolated(anon):0kB isolated(file):128kB mapped:4348kB dirty:0kB writeback:0kB shmem:1036kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 270336kB writeback_tmp:0kB kernel_stack:2208kB pagetables:3508kB all_unreclaimable? yes
[11514.036265] Node 0 DMA free:4212kB min:696kB low:868kB high:1040kB reserved_highatomic:0KB active_anon:0kB inactive_anon:9712kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15004kB managed:14368kB mlocked:0kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:0kB
[11514.036271] lowmem_reserve[]: 0 894 894 894 894
[11514.036275] Node 0 DMA32 free:46124kB min:44356kB low:55444kB high:66532kB reserved_highatomic:2048KB active_anon:1172kB inactive_anon:653564kB active_file:16kB inactive_file:1648kB unevictable:5584kB writepending:0kB present:1027952kB managed:974684kB mlocked:5584kB bounce:0kB free_pcp:1248kB local_pcp:764kB free_cma:0kB
[11514.036281] lowmem_reserve[]: 0 0 0 0 0
[11514.036284] Node 0 DMA: 3*4kB (E) 9*8kB (UME) 4*16kB (UME) 6*32kB (UME) 4*64kB (UME) 5*128kB (UME) 4*256kB (UME) 2*512kB (U) 1*1024kB (E) 0*2048kB 0*4096kB = 4308kB
[11514.036299] Node 0 DMA32: 68*4kB (ME) 93*8kB (UME) 69*16kB (UE) 49*32kB (UME) 22*64kB (UME) 12*128kB (UME) 5*256kB (UM) 29*512kB (M) 23*1024kB (UM) 0*2048kB 0*4096kB = 46312kB
[11514.036314] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[11514.036316] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[11514.036318] 1835 total pagecache pages
[11514.036319] 0 pages in swap cache
[11514.036319] Swap cache stats: add 0, delete 0, find 0/0
[11514.036321] Free swap = 0kB
[11514.036321] Total swap = 0kB
[11514.036322] 260739 pages RAM
[11514.036323] 0 pages HighMem/MovableOnly
[11514.036323] 13476 pages reserved
[11514.036324] 0 pages hwpoisoned
[11514.036325] Tasks state (memory values in pages):
[11514.036325] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[11514.036328] [ 349] 0 349 28900 805 225280 0 -250 systemd-journal
[11514.036333] [ 376] 0 376 5454 408 65536 0 -1000 systemd-udevd
[11514.036337] [ 558] 107 558 1777 459 53248 0 0 rpcbind
[11514.036340] [ 559] 0 559 624 128 45056 0 0 bpfilter_umh
[11514.036342] [ 560] 101 560 22493 568 69632 0 0 systemd-timesyn
[11514.036345] [ 600] 102 600 6645 232 73728 0 0 systemd-network
[11514.036347] [ 602] 103 602 5967 1003 86016 0 0 systemd-resolve
[11514.036350] [ 651] 105 651 1862 470 53248 0 -900 dbus-daemon
[11514.036352] [ 652] 0 652 1279 55 45056 0 0 iscsid
[11514.036355] [ 653] 0 653 1405 1357 49152 0 -17 iscsid
[11514.036376] [ 658] 0 658 6631 2561 86016 0 0 networkd-dispat
[11514.036379] [ 660] 584788 660 178782 1207 114688 0 0 updater
[11514.036382] [ 661] 584788 661 178945 1227 122880 0 0 agent
[11514.036384] [ 664] 0 664 4165 789 73728 0 0 systemd-logind
[11514.036386] [ 688] 0 688 3045 768 69632 0 -1000 sshd
[11514.036389] [ 714] 0 714 950 516 45056 0 0 atd
[11514.036391] [ 730] 0 730 717 395 40960 0 0 agetty
[11514.036393] [ 734] 0 734 1098 435 49152 0 0 agetty
[11514.036395] [ 741] 0 741 26287 2484 102400 0 0 unattended-upgr
[11514.036397] [ 782] 584788 782 179285 1863 126976 0 0 gomon
[11514.036400] [ 1535] 0 1535 218640 3823 249856 0 -900 snapd
[11514.036402] [ 2068] 0 2068 3449 872 65536 0 0 sshd
[11514.036405] [ 2071] 1001 2071 4594 994 73728 0 0 systemd
[11514.036407] [ 2072] 1001 2072 42119 797 94208 0 0 (sd-pam)
[11514.036409] [ 2158] 1001 2158 3482 779 65536 0 0 sshd
[11514.036411] [ 2159] 1001 2159 1529 612 45056 0 0 bash
[11514.036413] [ 2173] 1001 2173 370474 148369 1593344 0 0 db1000n
[11514.036415] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/session-3.scope,task=db1000n,pid=2173,uid=1001
[11514.036438] Out of memory: Killed process 2173 (db1000n) total-vm:1481896kB, anon-rss:593476kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:1556kB oom_score_adj:0
vitich
commented
2 years ago sudo sysctl -a | grep "vm.overcommit_memory"And if !=2 do
sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=80
seriyps
commented
1 year ago It looks like there is some kind of memory leak. This is how my RAM consumption graph over 24h looks
rkoten
commented
1 year ago Getting OOM killed on 0.9.20 running on a Raspberry Pi 3B+ with 1GB RAM. Before this version I used to run various releases from around May 2022 and never got this issue.
pashagolub
commented
1 year ago Probably the same issue on Windows causing OS to reboot
started with ulimit -n 65535
[101236.553584] app invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [101236.649900] CPU: 0 PID: 5641 Comm: app Kdump: loaded Not tainted 5.4.17-2136.307.3.4.el8uek.x86_64 #2 [101236.651576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021 [101236.749970] Call Trace: [101236.751757] dump_stack+0x6d/0x8b [101236.752512] dump_header+0x4f/0x1e1 [101236.848687] oom_kill_process.cold.33+0xb/0x10 [101236.852238] out_of_memory+0x1bf/0x551 [101236.948560] alloc_pages_slowpath+0xd40/0xe24 [101236.949738] alloc_pages_nodemask+0x2df/0x326 [101236.950962] alloc_pages_current+0x81/0xd6 [101236.952591] page_cache_alloc+0x6a/0x94 [101236.953354] pagecache_get_page+0xbe/0x2d9 [101237.048339] filemap_fault+0x7b5/0xa87 [101237.049089] ? mod_memcg_lruvec_state+0x27/0x102 [101237.050099] ? unlock_page_memcg+0x12/0x14 [101237.050857] ? _cond_resched+0x19/0x29 [101237.051543] ? down_read+0x12/0x98 [101237.052204] xfs_filemap_fault+0x6f/0x210 [xfs] [101237.053046] ? filemap_map_pages+0x28d/0x3a7 [101237.053849] xfs_filemap_fault+0x37/0x40 [xfs] [101237.148929] __do_fault+0x3c/0xd6 [101237.149586] handle_mm_fault+0xa71/0xd5d [101237.150355] handle_mm_fault+0xc9/0x1f0 [101237.151064] __do_page_fault+0x1f7/0x4b7 [101237.151791] do_page_fault+0x36/0x11a [101237.152499] do_async_page_fault+0x64/0xbe [101237.248368] ? prepare_exit_to_usermode+0xa5/0xce [101237.249245] async_page_fault+0x13d/0x142 [101237.249935] RIP: 0033:0x464a40 [101237.250536] Code: Bad RIP value. [101237.251116] RSP: 002b:000000c000047f30 EFLAGS: 00010202 [101237.252079] RAX: 0000000000002710 RBX: 0000000000002710 RCX: 0000000000004e20 [101237.253373] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005c12a210c7f5 [101237.348751] RBP: 000000c000047f90 R08: 0000000000016159 R09: 000000c000047f00 [101237.350025] R10: 0000000000000000 R11: 0000000000000212 R12: 000000c000047958 [101237.351335] R13: 000000c000038000 R14: 000000c0000024e0 R15: 00007f4462f98134 [101237.352680] Mem-Info: [101237.448297] active_anon:60964 inactive_anon:61225 isolated_anon:0 active_file:439 inactive_file:554 isolated_file:17 unevictable:3092 dirty:0 writeback:0 unstable:0 slab_reclaimable:6330 slab_unreclaimable:16484 mapped:3385 shmem:57 pagetables:3784 bounce:0 free:7784 free_pcp:343 free_cma:0 [101237.751974] Node 0 active_anon:243860kB inactive_anon:244900kB active_file:1264kB inactive_file:2716kB unevictable:12368kB isolated(anon):0kB isolated(file):0kB mapped:11972kB dirty:0kB writeback:0kB shmem:228kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2048kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [101237.952274] Node 0 DMA free:2876kB min:712kB low:888kB high:1064kB active_anon:4256kB inactive_anon:6220kB active_file:32kB inactive_file:104kB unevictable:0kB writepending:0kB present:15004kB managed:14372kB mlocked:0kB kernel_stack:64kB pagetables:196kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [101238.052569] lowmem_reserve[]: 0 563 563 563 563 [101238.149378] Node 0 DMA32 free:28436kB min:28620kB low:35772kB high:42924kB active_anon:239604kB inactive_anon:238680kB active_file:1744kB inactive_file:2876kB unevictable:12368kB writepending:0kB present:1027952kB managed:660636kB mlocked:12368kB kernel_stack:3904kB pagetables:14944kB bounce:0kB free_pcp:1240kB local_pcp:716kB free_cma:0kB [101238.348628] lowmem_reserve[]: 0 0 0 0 0 [101238.349898] Node 0 DMA: 624kB (UM) 318kB (UM) 2116kB (UM) 932kB (UM) 064kB 0128kB 1256kB (M) 1512kB (M) 11024kB (M) 02048kB 04096kB = 2912kB [101238.448614] Node 0 DMA32: 4354kB (UME) 4968kB (UME) 32116kB (UME) 9732kB (ME) 4064kB (UME) 27128kB (UME) 12256kB (UM) 8512kB (UM) 21024kB (UM) 02048kB 04096kB = 29180kB [101238.452727] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [101238.550119] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [101238.552333] 3818 total pagecache pages [101238.553539] 72 pages in swap cache [101238.649312] Swap cache stats: add 90806805, delete 90806621, find 2572123/47328139 [101238.651240] Free swap = 0kB [101238.652337] Total swap = 1397756kB [101238.653517] 260739 pages RAM [101238.748902] 0 pages HighMem/MovableOnly [101238.750411] 91987 pages reserved [101238.751559] 0 pages cma reserved [101238.752723] 0 pages hwpoisoned [101238.849090] Tasks state (memory values in pages): [101238.850776] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [101238.856406] [ 1042] 0 1042 26645 961 217088 280 0 systemd-journal [101238.950800] [ 1080] 0 1080 27270 823 225280 987 -1000 systemd-udevd [101239.049048] [ 1377] 32 1377 16805 964 172032 176 0 rpcbind [101239.051282] [ 1378] 0 1378 18749 388 147456 151 -1000 auditd [101239.148525] [ 1403] 996 1403 4404 359 65536 31 0 lsmd [101239.151268] [ 1404] 997 1404 408066 580 344064 2584 0 polkitd [101239.153828] [ 1405] 0 1405 12570 578 143360 265 0 smartd [101239.250177] [ 1408] 992 1408 13123 317 135168 132 0 chronyd [101239.252243] [ 1410] 81 1410 21283 835 167936 131 -900 dbus-daemon [101239.254408] [ 1412] 0 1412 100563 998 401408 365 0 sssd [101239.349668] [ 1418] 0 1418 31223 419 135168 130 0 irqbalance [101239.351845] [ 1448] 0 1448 131786 1284 520192 6036 0 firewalld [101239.353960] [ 1455] 0 1455 23584 991 204800 507 0 systemd-logind [101239.450418] [ 1496] 0 1496 150418 1278 389120 1069 0 NetworkManager [101239.452658] [ 1502] 0 1502 155985 1059 425984 3036 0 tuned [101239.549978] [ 1508] 0 1508 77894 393 176128 173 0 gssproxy [101239.552237] [ 1851] 0 1851 20996 1190 196608 497 0 dhclient [101239.649791] [ 1979] 0 1979 14345 3072 159744 0 -17 iscsid [101239.748291] [ 1980] 990 1980 309935 543 196608 2539 0 updater [101239.750463] [ 1982] 0 1982 59010 381 90112 43 0 OSWatcher [101239.752550] [ 1984] 991 1984 310101 690 208896 2964 0 agent [101239.754577] [ 1987] 0 1987 57421 487 225280 530 0 rsyslogd [101239.850123] [ 1994] 0 1994 23102 1076 204800 201 -1000 sshd [101239.948190] [ 2059] 0 2059 56498 339 69632 32 0 agetty [101239.953909] [ 2060] 0 2060 56588 307 69632 31 0 agetty [101240.051365] [ 2062] 0 2062 61687 293 114688 169 0 crond [101240.149554] [ 2064] 991 2064 328809 1331 233472 2364 0 gomon [101240.151524] [ 2067] 0 2067 11008 464 114688 73 0 atd [101240.249618] [ 2152] 0 2152 89327 711 311296 273 0 sudo [101240.251498] [ 2219] 0 2219 22377 757 212992 365 0 systemd [101240.348609] [ 2228] 0 2228 93053 1 311296 1375 0 (sd-pam) [101240.350696] [ 2254] 0 2254 2165 207 65536 38 0 osms-agent [101240.448427] [ 2257] 0 2257 392057 0 634880 22502 0 osms-agent [101240.451436] [ 2582] 0 2582 89327 675 303104 272 0 sudo [101240.454929] [ 2587] 989 2587 22350 846 204800 366 0 systemd [101240.549370] [ 2589] 989 2589 96642 1 315392 1383 0 (sd-pam) [101240.552724] [ 2595] 989 2595 310399 645 208896 2490 0 runcommand [101240.648889] [ 5361] 0 5361 41480 1324 327680 332 0 sshd [101240.754820] [ 5367] 1000 5367 22379 770 212992 372 0 systemd [101240.849961] [ 5370] 1000 5370 133508 1 331776 1393 0 (sd-pam) [101240.851812] [ 5376] 1000 5376 41536 614 311296 378 0 sshd [101240.949089] [ 5377] 1000 5377 58857 428 98304 481 0 bash [101240.951001] [ 5515] 1000 5515 58857 65 77824 481 0 bash [101241.051001] [ 5522] 1000 5522 473989 59914 2592768 168092 0 db1000n [101241.149031] [ 5633] 1000 5633 177109 828 143360 1523 0 launcher-disbal [101241.248209] [ 5640] 1000 5640 177639 661 118784 1264 0 app [101241.348277] [ 332172] 0 332172 360142 55754 2113536 116225 0 dnf [101241.453130] [ 333542] 0 333542 23450 414 204800 18 0 sssd_nss [101241.549421] [ 333546] 0 333546 58944 391 90112 12 0 nfssub [101241.653268] [ 333560] 0 333560 25050 433 217088 19 0 sssd_be [101241.655693] [ 333565] 0 333565 60051 800 106496 54 0 nfsiostat [101241.749907] [ 333605] 0 333605 18564 571 180224 15 0 sshd [101241.753146] [ 333615] 0 333615 18564 517 176128 7 0 sshd [101241.850937] [ 333619] 0 333619 68537 598 147456 172 0 crond [101241.948481] [ 333632] 0 333632 68537 528 147456 158 0 crond [101242.049578] [ 333635] 0 333635 55630 266 77824 1 0 sh [101242.051493] [ 333639] 0 333639 55630 55 69632 2 0 sh [101242.149457] [ 333641] 0 333641 59952 699 106496 16 0 ksplice [101242.151602] [ 333646] 0 333646 58944 326 90112 0 0 oswnet [101242.250548] [ 333653] 0 333653 18564 480 172032 0 0 sshd [101242.350970] [ 333656] 0 333656 58944 290 90112 0 0 iosub [101242.352850] [ 333658] 0 333658 15206 158 143360 0 0 sshd [101242.449099] [ 333659] 0 333659 58944 68 73728 0 0 oswnet [101242.450996] [ 333660] 0 333660 68537 94 131072 152 0 crond [101242.452898] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=db1000n,pid=5522,uid=1000 [101242.649253] Out of memory: Killed process 5522 (db1000n) total-vm:1895956kB, anon-rss:239656kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2532kB oom_score_adj:0 [101243.552711] oom_reaper: reaped process 5522 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [417432.812358] TCP: too many orphaned sockets