search

CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.3k stars 282 forks source link

ARDB crashes all the time #474

Closed Zakaren closed 1 year ago

Zakaren commented 4 years ago

Hi ! I'm using AIL since two months, and ARDB is really unstable, it crashes all the time, and i'm forced to restart all the scripts again via ./LAUNCH -l to make AIL work properly again.

The last logs I received from ARDB : 

...
[1335] 02-20 23:20:00,310 INFO slave_sync_instantaneous_ops_per_sec:0
[1335] 02-20 23:20:00,310 INFO total_commands_processed:1404149
[1335] 02-20 23:20:00,310 INFO instantaneous_ops_per_sec:37
[1335] 02-20 23:20:00,310 INFO total_connections_received:1511
[1335] 02-20 23:20:00,310 INFO rejected_connections:0
[1335] 02-20 23:20:00,310 INFO ========================Period Statistics Dump End===========================
bash: line 1:  1335 Killed                  ardb-server /home/debian/AIL-framework//configs/6382.conf

Have you an idea why it crashes all the time ? Thank you,

Terrtia commented 4 years ago

Hi @Zakaren !

We can try to enable ARDB logfile. Can you please edit configs/6382.conf as follow:

A log file should be created in DATA_ARDB/log/

We can check this file if you have another crash.

Zakaren commented 4 years ago

Hi @Terrtia !

Thank you for your reply,

I edited the configs/6382.conf file as you described, below the logs produced during the last crash of ARDB : 

[22721] 02-24 14:20:00,523 INFO ========================Period Statistics Dump Begin===========================
[22721] 02-24 14:20:00,523 INFO coststat_select_all:calls=7,costs=8,cost_per_call=1,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_select_range[0-1000]:calls=7,costs=8,cost_per_call=1,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_get_all:calls=487,costs=20154,cost_per_call=41,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_get_range[0-1000]:calls=487,costs=20154,cost_per_call=41,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_set_all:calls=36,costs=2603,cost_per_call=72,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_set_range[0-1000]:calls=36,costs=2603,cost_per_call=72,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_incrby_all:calls=165,costs=13948,cost_per_call=84,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_incrby_range[0-1000]:calls=165,costs=13948,cost_per_call=84,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_hexists_all:calls=156,costs=10706,cost_per_call=68,percents=100.0000%
[22721] 02-24 14:20:00,523 INFO coststat_hexists_range[0-1000]:calls=155,costs=7723,cost_per_call=49,percents=99.3590%
[22721] 02-24 14:20:00,523 INFO coststat_hexists_range[1000-5000]:calls=1,costs=2983,cost_per_call=2983,percents=0.6410%
[22721] 02-24 14:20:00,523 INFO coststat_hget_all:calls=22,costs=1073,cost_per_call=48,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hget_range[0-1000]:calls=22,costs=1073,cost_per_call=48,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hincrby_all:calls=2984,costs=373381,cost_per_call=125,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hincrby_range[0-1000]:calls=2982,costs=370569,cost_per_call=124,percents=99.9330%
[22721] 02-24 14:20:00,524 INFO coststat_hincrby_range[1000-5000]:calls=2,costs=2812,cost_per_call=1406,percents=0.0670%
[22721] 02-24 14:20:00,524 INFO coststat_hincrbyfloat_all:calls=274,costs=25523,cost_per_call=93,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hincrbyfloat_range[0-1000]:calls=274,costs=25523,cost_per_call=93,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hset_all:calls=676,costs=41536,cost_per_call=61,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_hset_range[0-1000]:calls=676,costs=41536,cost_per_call=61,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_scard_all:calls=800,costs=37759,cost_per_call=47,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_scard_range[0-1000]:calls=800,costs=37759,cost_per_call=47,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_sadd_all:calls=677,costs=86706,cost_per_call=128,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_sadd_range[0-1000]:calls=676,costs=85228,cost_per_call=126,percents=99.8523%
[22721] 02-24 14:20:00,524 INFO coststat_sadd_range[1000-5000]:calls=1,costs=1478,cost_per_call=1478,percents=0.1477%
[22721] 02-24 14:20:00,524 INFO coststat_sismember_all:calls=11,costs=672,cost_per_call=61,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_sismember_range[0-1000]:calls=11,costs=672,cost_per_call=61,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_smembers_all:calls=54,costs=545059,cost_per_call=10093,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_smembers_range[0-1000]:calls=18,costs=8460,cost_per_call=470,percents=33.3333%
[22721] 02-24 14:20:00,524 INFO coststat_smembers_range[5000-10000]:calls=12,costs=98743,cost_per_call=8228,percents=22.2222%
[22721] 02-24 14:20:00,524 INFO coststat_smembers_range[10000-20000]:calls=16,costs=230920,cost_per_call=14432,percents=29.6296%
[22721] 02-24 14:20:00,524 INFO coststat_smembers_range[20000-50000]:calls=8,costs=206936,cost_per_call=25867,percents=14.8148%
[22721] 02-24 14:20:00,524 INFO coststat_zadd_all:calls=822,costs=90395,cost_per_call=109,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zadd_range[0-1000]:calls=822,costs=90395,cost_per_call=109,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zcard_all:calls=698,costs=18338,cost_per_call=26,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zcard_range[0-1000]:calls=698,costs=18338,cost_per_call=26,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zincrby_all:calls=780,costs=93245,cost_per_call=119,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zincrby_range[0-1000]:calls=779,costs=81217,cost_per_call=104,percents=99.8718%
[22721] 02-24 14:20:00,524 INFO coststat_zincrby_range[10000-20000]:calls=1,costs=12028,cost_per_call=12028,percents=0.1282%
[22721] 02-24 14:20:00,524 INFO coststat_zrange_all:calls=150,costs=3325,cost_per_call=22,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zrange_range[0-1000]:calls=150,costs=3325,cost_per_call=22,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zscore_all:calls=932,costs=28658,cost_per_call=30,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO coststat_zscore_range[0-1000]:calls=932,costs=28658,cost_per_call=30,percents=100.0000%
[22721] 02-24 14:20:00,524 INFO slave_sync_total_commands_processed:0
[22721] 02-24 14:20:00,524 INFO slave_sync_instantaneous_ops_per_sec:0
[22721] 02-24 14:20:00,524 INFO total_commands_processed:3033475
[22721] 02-24 14:20:00,524 INFO instantaneous_ops_per_sec:21
[22721] 02-24 14:20:00,524 INFO total_connections_received:1394
[22721] 02-24 14:20:00,524 INFO rejected_connections:0
[22721] 02-24 14:20:00,524 INFO ========================Period Statistics Dump End===========================

Thanks in advance,

Terrtia commented 4 years ago

Hi @Zakaren !

Can you please set the loglevel to debug ? (edit L110)

Zakaren commented 4 years ago

Hi @Terrtia,

It seems that it's the kernel who is killing ARDB. Below the kern.log logs : 

Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.477047] **ardb-server invoked oom-killer**: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.479219] ardb-server cpuset=/ mems_allowed=0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.480400] CPU: 0 PID: 22443 Comm: ardb-server Not tainted 4.9.0-12-amd64 #1 Debian 4.9.210-1
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.481828] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 2:1.10.2-58953eb7 04/01/2014
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.483328]  0000000000000000 ffffffffad136bfe ffffb5778164fc10 ffff9669e7d30080
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.484893]  ffffffffad009adb 0000000000000000 0000000000000000 0000000000010286
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.486288]  ffffb5778164fab0 0000000000000000 ffffffffc0425765 ffffffffad0ad670
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.487793] Call Trace:
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.490118]  [<ffffffffad136bfe>] ? dump_stack+0x66/0x88
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.491625]  [<ffffffffad009adb>] ? dump_header+0x78/0x1fd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.493687]  [<ffffffffc0425765>] ? virtballoon_oom_notify+0x25/0x70 [virtio_balloon]
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.495421]  [<ffffffffad0ad670>] ? cap_inode_killpriv+0x20/0x20
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.497400]  [<ffffffffacf894da>] ? oom_kill_process+0x22a/0x3f0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.499470]  [<ffffffffacf89971>] ? out_of_memory+0x111/0x470
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.501355]  [<ffffffffacf8ea6f>] ? __alloc_pages_slowpath+0xa1f/0xb30
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.503493]  [<ffffffffacf8ed81>] ? __alloc_pages_nodemask+0x201/0x260
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.505574]  [<ffffffffacfe0871>] ? alloc_pages_current+0x91/0x140
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.506671]  [<ffffffffacf87906>] ? filemap_fault+0x326/0x5d0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.510528]  [<ffffffffc0320bc1>] ? ext4_filemap_fault+0x31/0x50 [ext4]
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.511928]  [<ffffffffacfb8cd9>] ? __do_fault+0xa9/0x1c0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.512962]  [<ffffffffacfbced4>] ? handle_mm_fault+0xda4/0x1350
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.514563]  [<fffffffface2568f>] ? __switch_to+0x1ef/0x670
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.516316]  [<ffffffffad41e2f1>] ? __switch_to_asm+0x41/0x70
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.517983]  [<fffffffface63475>] ? __do_page_fault+0x255/0x4f0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.519208]  [<ffffffffad41f618>] ? async_page_fault+0x28/0x30
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.520435] Mem-Info:
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283] active_anon:1904528 inactive_anon:9374 isolated_anon:0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283]  active_file:31 inactive_file:75 isolated_file:0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283]  unevictable:0 dirty:0 writeback:0 unstable:0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283]  slab_reclaimable:3016 slab_unreclaimable:30140
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283]  mapped:2120 shmem:10396 pagetables:11518 bounce:0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.522283]  free:25724 free_pcp:29 free_cma:0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.534536] Node 0 active_anon:7618112kB inactive_anon:37496kB active_file:124kB inactive_file:600kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8480kB dirty:0kB writeback:0kB shmem:41584kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:3866 all_unreclaimable? no
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.541607] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.547850] lowmem_reserve[]: 0 2973 7759 7759 7759
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.548917] Node 0 DMA32 free:44844kB min:25848kB low:32308kB high:38768kB active_anon:2976872kB inactive_anon:8380kB active_file:24kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129200kB managed:3063632kB mlocked:0kB slab_reclaimable:436kB slab_unreclaimable:23284kB kernel_stack:96kB pagetables:7680kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.554422] lowmem_reserve[]: 0 0 4785 4785 4785
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.555357] Node 0 Normal free:41676kB min:41596kB low:51992kB high:62388kB active_anon:4641240kB inactive_anon:29116kB active_file:324kB inactive_file:376kB unevictable:0kB writepending:0kB present:5046272kB managed:4905392kB mlocked:0kB slab_reclaimable:11628kB slab_unreclaimable:97276kB kernel_stack:4864kB pagetables:38392kB bounce:0kB free_pcp:72kB local_pcp:0kB free_cma:0kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.571078] lowmem_reserve[]: 0 0 0 0 0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.572696] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.579381] Node 0 DMA32: 276*4kB (UME) 449*8kB (UME) 620*16kB (UME) 275*32kB (UME) 156*64kB (UME) 57*128kB (UE) 13*256kB (UME) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 45048kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.586060] Node 0 Normal: 1402*4kB (UMEH) 1357*8kB (UMEH) 1308*16kB (UMEH) 106*32kB (UMEH) 8*64kB (MEH) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41552kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.589395] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.591259] 10614 total pagecache pages
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.592166] 0 pages in swap cache
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.593128] Swap cache stats: add 0, delete 0, find 0/0
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.594572] Free swap  = 0kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.595248] Total swap = 0kB
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.596467] 2047866 pages RAM
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.596989] 0 pages HighMem/MovableOnly
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.597633] 51633 pages reserved
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.598160] 0 pages hwpoisoned
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.598666] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.600054] [  495]     0   495    17670     2160      30       3        0             0 systemd-journal
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.601571] [  581]     0   581    11617      430      24       3        0         -1000 systemd-udevd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.603132] [  718]   100   718    31821      133      32       3        0             0 systemd-timesyn
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.604767] [  801]     0   801     5120      259      11       3        0             0 dhclient
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.606217] [  830]     0   830     5120      260      12       3        0             0 dhclient
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.607629] [  930]     0   930     9495      124      22       3        0             0 systemd-logind
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.609261] [  931]     0   931     7409       62      19       3        0             0 cron
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.612596] [  935]     0   935    62528      287      28       3        0             0 rsyslogd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.618298] [  936]   106   936    11282      110      27       3        0          -900 dbus-daemon
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.624730] [  954]     0   954    17489      192      38       3        0         -1000 sshd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.629430] [  957]     0   957     8604      205      19       3        0             0 nginx
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.633579] [  959]     0   959     3631       33      12       3        0             0 agetty
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.636982] [  960]     0   960     3575       32      12       3        0             0 agetty
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.639863] [  961]    33   961     8819      434      22       3        0             0 nginx
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.642608] [  962]    33   962     8763      359      21       3        0             0 nginx
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.645592] [  968]   107   968    21800     6259      48       4        0             0 tor
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.649007] [22102]  1000 22102     7078      247      17       3        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.653753] [22104]  1000 22104     5012      137      14       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.657369] [22109]  1000 22109     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.660334] [22110]  1000 22110   109497    38413     164       3        0             0 redis-server
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.663196] [22116]  1000 22116     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.665809] [22117]  1000 22117    25529      461      44       3        0             0 redis-server
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.667680] [22123]  1000 22123     2795       56       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.669280] [22124]  1000 22124     6994      122      18       3        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.671088] [22126]  1000 22126   161209    35172     203       3        0             0 redis-server
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.672818] [22127]  1000 22127     5012      137      15       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.674479] [22135]  1000 22135     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.676063] [22137]  1000 22137     7036      164      18       3        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.677487] [22139]  1000 22139   705208   307780    1055       6        0             0 ardb-server
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.679055] [22140]  1000 22140     5012      137      14       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.680551] [22165]  1000 22165     2796       54       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.681999] [22166]  1000 22166    16172     2708      34       3        0             0 log_subscriber
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.683596] [22168]  1000 22168     2796       55       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.685156] [22170]  1000 22170    16191     2722      35       3        0             0 log_subscriber
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.686780] [22171]  1000 22171     6994      122      18       3        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.688282] [22173]  1000 22173     5012      136      13       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.689688] [22178]  1000 22178     2795       56       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.691155] [22179]  1000 22179     7223     1267      18       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.692621] [22182]  1000 22182    50921     9511      72       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.694188] [22183]  1000 22183    29370     7209      60       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.695634] [22184]  1000 22184    33804    11542      71       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.697118] [22185]  1000 22185    24874     2692      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.698541] [22186]  1000 22186    24888     2687      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.700018] [22188]  1000 22188    24872     2688      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.701490] [22190]  1000 22190    24873     2690      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.702959] [22192]  1000 22192    24888     2688      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.704429] [22194]  1000 22194    24872     2689      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.705849] [22195]  1000 22195    24888     2684      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.707313] [22196]  1000 22196    24872     2689      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.708806] [22197]  1000 22197    24887     2683      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.710251] [22198]  1000 22198    24873     2690      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.711821] [22199]  1000 22199    24875     2689      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.713200] [22200]  1000 22200    24872     2687      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.714584] [22201]  1000 22201    24895     2716      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.715989] [22202]  1000 22202    24889     2689      54       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.717427] [22203]  1000 22203    24910     2696      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.718941] [22204]  1000 22204    24891     2691      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.720560] [22205]  1000 22205    24875     2691      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.722154] [22206]  1000 22206    24888     2688      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.723665] [22207]  1000 22207    24879     2697      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.725403] [22208]  1000 22208    24889     2689      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.727092] [22209]  1000 22209    45386     2708      55       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.728845] [22210]  1000 22210    24873     2690      52       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.730441] [22212]  1000 22212    24888     2689      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.731972] [22214]  1000 22214    45376     2701      56       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.733494] [22215]  1000 22215    24889     2689      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.734988] [22220]  1000 22220    24873     2688      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.736422] [22221]  1000 22221    24890     2690      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.737901] [22223]  1000 22223    24891     2691      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.740181] [22224]  1000 22224    24875     2690      50       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.744823] [22225]  1000 22225    24878     2696      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.750066] [22227]  1000 22227    24872     2689      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.755222] [22228]  1000 22228    24888     2686      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.759176] [22229]  1000 22229    24889     2689      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.762942] [22231]  1000 22231    24875     2691      51       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.766770] [22233]  1000 22233    24875     2690      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.769368] [22235]  1000 22235    24889     2687      52       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.771976] [22236]  1000 22236    24910     2694      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.778437] [22237]  1000 22237    24873     2688      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.782413] [22238]  1000 22238    24874     2689      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.786315] [22239]  1000 22239    24888     2689      50       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.789095] [22240]  1000 22240    24894     2696      51       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.792180] [22241]  1000 22241    24873     2689      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.794896] [22242]  1000 22242    24879     2694      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.797570] [22243]  1000 22243    24872     2687      50       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.799408] [22244]  1000 22244    24891     2691      51       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.801163] [22245]  1000 22245    24876     2692      55       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.802898] [22246]  1000 22246    24875     2691      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.804661] [22247]  1000 22247    24872     2688      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.806442] [22248]  1000 22248    24894     2695      52       4        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.808059] [22250]  1000 22250    24875     2691      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.809456] [22251]  1000 22251    24891     2691      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.810854] [22254]  1000 22254    24887     2684      54       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.812300] [22255]  1000 22255    24889     2689      49       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.813703] [22256]  1000 22256    24875     2691      53       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.815164] [22257]  1000 22257    24875     2691      52       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.817041] [22258]  1000 22258    24891     2689      50       3        0             0 python3
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.818517] [22472]  1000 22472     8424     1597      19       4        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.819979] [22474]  1000 22474     5012      136      14       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.821390] [22479]  1000 22479     2796       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.822857] [22480]  1000 22480    73785    11219     116       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.824342] [22482]  1000 22482     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.825833] [22484]  1000 22484    38293    16090      80       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.827265] [22486]  1000 22486     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.828759] [22488]  1000 22488    41680    19437      88       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.830482] [22491]  1000 22491     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.832206] [22493]  1000 22493    95915    24809     158       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.833602] [22499]  1000 22499     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.834985] [22500]  1000 22500   102246    29165     167       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.836445] [22502]  1000 22502     2795       54       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.837940] [22504]  1000 22504   101877    25462     158       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.839586] [22507]  1000 22507     2795       57       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.841144] [22508]  1000 22508    94204    24110     152       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.842673] [22510]  1000 22510     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.844131] [22512]  1000 22512    57130    28732     116       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.845607] [22517]  1000 22517     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.847126] [22519]  1000 22519    96621    24908     157       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.848712] [22521]  1000 22521     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.850188] [22524]  1000 22524    96756    25027     155       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.851716] [22528]  1000 22528     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.853245] [22531]  1000 22531    94199    24109     152       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.854737] [22535]  1000 22535     2795       57       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.856294] [22538]  1000 22538   101419    27586     168       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.857766] [22542]  1000 22542     2795       56      11       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.859250] [22543]  1000 22543    96467    24716     156       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.861065] [22546]  1000 22546     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.862518] [22548]  1000 22548   137182    68451     237       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.864158] [22552]  1000 22552     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.865761] [22554]  1000 22554   101857    32153     169       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.867273] [22559]  1000 22559     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.868791] [22564]  1000 22564     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.870252] [22565]  1000 22565   254089   146075     393       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.871681] [22569]  1000 22569     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.873107] [22571]  1000 22571    89200    16497     144       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.874549] [22574]  1000 22574     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.875934] [22576]  1000 22576    54226    24693     108       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.877315] [22579]  1000 22579     2795       54       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.878857] [22581]  1000 22581    56569    25785     114       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.880373] [22584]  1000 22584     2795       56       8       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.881879] [22587]  1000 22587    56744    25824     114       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.883293] [22591]  1000 22591     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.884712] [22592]  1000 22592   254394   141627     385       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.886202] [22596]  1000 22596     2795       54       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.887649] [22597]  1000 22597   119207    32738     172       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.889121] [22601]  1000 22601     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.890575] [22602]  1000 22602    93922    23765     150       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.892064] [22608]  1000 22608     2795       54      11       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.893503] [22609]  1000 22609    28882     3957      61       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.894860] [22614]  1000 22614     2795       54       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.896319] [22615]  1000 22615   298607   184026     464       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.898025] [22618]  1000 22618     2796       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.899398] [22619]  1000 22619    80881    11877     132       5        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.900801] [22624]  1000 22624     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.902327] [22625]  1000 22625    84295    15824     139       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.903771] [22628]  1000 22628     2796       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.905136] [22630]  1000 22630    86854    14284     141       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.906702] [22633]  1000 22633     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.908226] [22635]  1000 22635    37666     9275      77       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.912736] [22638]  1000 22638     2795       54      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.917605] [22639]  1000 22639   301833   188628     478       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.921201] [22644]  1000 22644     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.924467] [22650]  1000 22650     2795       56      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.927634] [22660]  1000 22660     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.929709] [22661]  1000 22661    40002     9926      80       3        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.932331] [22663]  1000 22663     2795       55      10       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.936251] [22665]  1000 22665     6994      163      17       3        0             0 screen
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.939636] [22666]  1000 22666     5012      137      14       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.942638] [22672]  1000 22672     2795       56       9       3        0             0 bash
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.945552] [22675]  1000 22675   190193    38525     193       4        0             0 python
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.948203] [13170]   108 13170    18854      185      41       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.950425] [13175]   108 13175    18854      212      39       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.952271] [13176]   108 13176    18854      221      40       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.954298] [13177]   108 13177    18854      221      40       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.956049] [13178]   108 13178    18854      221      40       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.957686] [13179]   108 13179    18854      203      40       3        0             0 zabbix_agentd
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.959364] Out of memory: Kill process 22139 (ardb-server) score 154 or sacrifice child
Mar  2 23:28:34 pp-dataleak-01 kernel: [380670.961498] Killed process 22139 (ardb-server) total-vm:2820832kB, anon-rss:1231120kB, file-rss:0kB, shmem-rss:0kB
Mar  2 23:28:36 pp-dataleak-01 kernel: [380672.834205] oom_reaper: reaped process 22139 (ardb-server), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Thanks in advance,

Terrtia commented 1 year ago

Fixed in AIL v5.0 The database has been migrated from ARDB to Kvrocks.