Live Monitoring still doesn't work

[UweLoyal]

[SylvainLegrand]

I found a similar problem : I use webmin on 5 servers. For 4 of them the new version works properly.

On the last one the real time monitoring page crash with an out of memory !

• VPS
• ubuntu 20.04.6
• Ram 7.57G (1.89G used)

If I refresh the page the memory consumed by webmin increase until the crash (I use htop) the command is "/usr/share/webmin/authentic-theme/sysinfo.cgi"

Greetings, Sylvain

[jcameron]

@iliajie please take a look at this ASAP

[iliajie]

For 4 of them the new version works properly.

On the last one the real time monitoring page crash with an out of memory !

I wonder what makes those servers different? Can you provide more debug information? Is there anything useful logged under /var/webmin and /var/webmin/modules/authentic-theme? Also what is logged in browser's console for that server that doesn't work?

If I refresh the page the memory consumed by webmin increase until the crash (I use htop)

I have run numerous tests and never had this issue. Can you please tell more about the system which is failing? How is it different in terms of configuration?

[SylvainLegrand]

I found something very strange : On /var/webmin/modules/authentic-theme the files stats-server-556.log, stats-server-557.log, stats-server-558.log, stats-server-559.log, and stats-server-560.log contain only one line : "Failed to run /usr/share/webmin/authentic-theme/stats.pl : No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."

But the file /usr/share/webmin/authentic-theme/stats.pl exist with chmod = 755 and chown = root:root

[SylvainLegrand]

Also I use a reverse proxy to connect to webmin. But is like my other servers.

[iliajie]

No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."

This is an odd one! Do you have /etc/webmin/authentic-theme directory?

[SylvainLegrand]

yes

[iliajie]

No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."

This is an odd one! Do you have /etc/webmin/authentic-theme directory?

Oh, I think I see. It's the PERLLIB or $root isn't set correctly.

What is the output of:

grep -iE 'PERLLIB|root' /etc/webmin/config /etc/webmin/miniserv.conf

[SylvainLegrand]

/etc/webmin/miniserv.conf:root=/usr/share/webmin /etc/webmin/miniserv.conf:preroot=authentic-theme

[iliajie]

/etc/webmin/authentic-theme/stats.pl

That looks fine!

What is the content of /etc/webmin/authentic-theme/stats.pl file?

[SylvainLegrand]

#!/usr/bin/perl
open(CONF, "</etc/webmin/miniserv.conf") || die "Failed to open /etc/webmin/miniserv.conf : $!";
while(<CONF>) {
        $root = $1 if (/^root=(.*)/);
        }
close(CONF);
$root || die "No root= line found in /etc/webmin/miniserv.conf";
$ENV{'PERLLIB'} = "$root";
$ENV{'WEBMIN_CONFIG'} = "/etc/webmin";
$ENV{'WEBMIN_VAR'} = "/var/webmin";
delete($ENV{'MINISERV_CONFIG'});
chdir("$root/authentic-theme");
exec("$root/authentic-theme/stats.pl", @ARGV) || die "Failed to run $root/authentic-theme/stats.pl : $!";

[iliajie]

That's odd. I don't understand the problem.

Is this Webmin instance running via a proxy? If so, do you experience the same issue when connected directly?

[jcameron]

Does /usr/bin/perl exist on your system?

[jcameron]

Also, does /usr/share/webmin/authentic-theme/stats.pl exist and what's the first line of that file?

[SylvainLegrand]

/usr/bin/perl does exist

/usr/share/webmin/authentic-theme/stats.pl does exist and the first line is #!/usr/local/bin/perl

[SylvainLegrand]

/usr/local/bin/perl doesn't exist !!!!!

[iliajie]

and the first line is #!/usr/local/bin/perl

Oh, no! But why? Jamie, Webmin should replace it, right?

[jcameron]

Oh shit there's the problem ... let me see what's up with that

[jcameron]

How was Webmin installed on this system exactly?

[SylvainLegrand]

Some years ago I do : `sudo wget https://download.webmin.com/jcameron-key.asc

sudo apt-key add jcameron-key.asc

sudo add-apt-repository 'deb http://download.webmin.com/download/repository sarge contrib'

sudo apt-get update

sudo apt-get install webmin -y ` Until now the updates works properly.

[iliajie]

But we do replace /usr/local/bin/perl with /usr/bin/perl.

What is the output of:

ls -lsa /usr/local/bin/perl

[SylvainLegrand]

ls: cannot access '/usr/local/bin/perl': No such file or directory

[jcameron]

Do any other files in /usr/share/webmin/authentic-theme contain /usr/local/bin/perl ?

[iliajie]

Does it fix the problem if you:

ln -s /usr/bin/perl /usr/local/bin/perl

[iliajie]

Do any other files in /usr/share/webmin/authentic-theme contain /usr/local/bin/perl ?

Or any other Webmin modules, for example acl/?

[SylvainLegrand]

all the files on /usr/share/webmin/authentic-theme contain /usr/local/bin/perl

[jcameron]

How about other dirs like /usr/share/webmin/acl ?

[iliajie]

all the files on /usr/share/webmin/authentic-theme contain /usr/local/bin/perl

Did you upgraded theme from the repo, using Theme Configuration page?

[SylvainLegrand]

yes, of course !

[iliajie]

And you did this after Webmin 2.201 was installed?

[SylvainLegrand]

files on acl folder only contain #!/usr/bin/perl

And you did this after Webmin 2.201 was installed? YES

[iliajie]

And you did this after Webmin 2.201 was installed? YES

That explains it! I will address it! This isn't hard to do! And we don't need a new Webmin release, which is a relief ..

[iliajie]

Does it fix the problem if you:

ln -s /usr/bin/perl /usr/local/bin/perl

You can work around it by doing this.

[SylvainLegrand]

Does it fix the problem if you:

ln -s /usr/bin/perl /usr/local/bin/perl

You can work around it by doing this.

It doesn't work !

[jcameron]

Best thing would be to re-install Webmin and over-write any changed files with apt-get reinstall webmin

[iliajie]

This issue with shebang was addressed in this patch https://github.com/webmin/authentic-theme/commit/24464c9dbfc3122d25334703927ef4b15fa7df80.

[SylvainLegrand]

Ok. I re-installed webmin :

1. apt remove
2. apt purge
3. delete all webmin directories
4. apt install

on authentic-theme files, now the right path is writing The log on webmin/modules/authentic-theme is now stats-server-555.log It's content :

[25/Jul/2024:00:22:14 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 555
[25/Jul/2024:00:22:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 50640 opened
[25/Jul/2024:00:22:20 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 50640 closed
[25/Jul/2024:00:22:20 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56568 opened
[25/Jul/2024:00:22:22 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56568 closed
[25/Jul/2024:00:22:22 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56578 opened
[25/Jul/2024:00:22:24 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56578 closed
[25/Jul/2024:00:22:24 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 opened
[25/Jul/2024:00:22:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 closed
[25/Jul/2024:00:22:39 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 is closed due to inactivity
[25/Jul/2024:00:34:40 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39210 opened
[25/Jul/2024:00:34:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39210 closed
[25/Jul/2024:00:34:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39214 opened
[25/Jul/2024:00:34:44 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39214 closed
[25/Jul/2024:00:34:44 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39226 opened
[25/Jul/2024:00:34:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39226 closed
[25/Jul/2024:00:34:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39234 opened
[25/Jul/2024:00:34:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39234 closed
[25/Jul/2024:00:34:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 opened
[25/Jul/2024:00:34:50 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 closed
[25/Jul/2024:00:35:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 is closed due to inactivity

But I face the same problem ! the command /usr/share/webmin/authentic-theme/sysinfo.cgi crash with an out of memory ! And a second command crash /usr/share/webmin/authentic-theme/xhr.cgi

The result !

[SylvainLegrand]

If I disable the reverse proxy I can connect thru the IP address (custom port - not 10000) but webmin behavior is the same ! No stats page !

[iliajie]

What if you force reload Webmin by running:

/etc/webmin/restart-by-force-kill

[SylvainLegrand]

Still the same problem

[Rafal-Bierowiec]

This issue with shebang was addressed in this patch webmin/authentic-theme@24464c9.

I had exactly the sam problem after upgrade to latest webmin version (websocket connection error) and downgrading authentic-theme by running following commands solved issue for me.

cd /usr/share/webmin/authentic-theme
./theme-update.sh --release:21.10 --force

[UweLoyal]

Very good Idea, now also works fine on my System!

stats.cgi call looks difference!

[iliajie]

If I disable the reverse proxy I can connect thru the IP address (custom port - not 10000) but webmin behavior is the same !

I really wish I could help, but sadly, I cannot reproduce this issue and can't understand what's causing it on your end either.

Perhaps, for now, the right solution would be to disable live monitoring on the Theme Configuration page rather than downgrading the theme.

[SylvainLegrand]

Nothing can resolve it !!!!

This is my syslog about the memory crash

Jul 31 16:23:31 infras kernel: [577035.822644] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jul 31 16:23:31 infras kernel: [577035.822648] CPU: 3 PID: 1 Comm: systemd Not tainted 5.4.0-189-generic #209-Ubuntu
Jul 31 16:23:31 infras kernel: [577035.822649] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 2:1.10.2-58953eb7 04/01/2014
Jul 31 16:23:31 infras kernel: [577035.822650] Call Trace:
Jul 31 16:23:31 infras kernel: [577035.822658]  dump_stack+0x6d/0x8b
Jul 31 16:23:31 infras kernel: [577035.822662]  dump_header+0x4f/0x1eb
Jul 31 16:23:31 infras kernel: [577035.822664]  oom_kill_process.cold+0xb/0x10
Jul 31 16:23:31 infras kernel: [577035.822668]  out_of_memory+0x1cf/0x500
Jul 31 16:23:31 infras kernel: [577035.822672]  __alloc_pages_slowpath+0xe14/0xee0
Jul 31 16:23:31 infras kernel: [577035.822675]  __alloc_pages_nodemask+0x2d0/0x320
Jul 31 16:23:31 infras kernel: [577035.822678]  alloc_pages_current+0x87/0xe0
Jul 31 16:23:31 infras kernel: [577035.822682]  __page_cache_alloc+0x72/0x90
Jul 31 16:23:31 infras kernel: [577035.822685]  pagecache_get_page+0xbf/0x300
Jul 31 16:23:31 infras kernel: [577035.822687]  filemap_fault+0x79e/0xbc0
Jul 31 16:23:31 infras kernel: [577035.822690]  ? unlock_page_memcg+0x12/0x20
Jul 31 16:23:31 infras kernel: [577035.822693]  ? page_add_file_rmap+0xff/0x1a0
Jul 31 16:23:31 infras kernel: [577035.822696]  ? xas_load+0xd/0x80
Jul 31 16:23:31 infras kernel: [577035.822699]  ? xas_find+0x17f/0x1c0
Jul 31 16:23:31 infras kernel: [577035.822701]  ? filemap_map_pages+0x24c/0x380
Jul 31 16:23:31 infras kernel: [577035.822705]  ext4_filemap_fault+0x32/0x50
Jul 31 16:23:31 infras kernel: [577035.822708]  __do_fault+0x3c/0x170
Jul 31 16:23:31 infras kernel: [577035.822710]  do_fault+0x24b/0x640
Jul 31 16:23:31 infras kernel: [577035.822712]  __handle_mm_fault+0x4c5/0x7a0
Jul 31 16:23:31 infras kernel: [577035.822715]  handle_mm_fault+0xca/0x200
Jul 31 16:23:31 infras kernel: [577035.822719]  do_user_addr_fault+0x1f9/0x450
Jul 31 16:23:31 infras kernel: [577035.822721]  __do_page_fault+0x58/0x90
Jul 31 16:23:31 infras kernel: [577035.822725]  ? exit_to_usermode_loop+0xea/0x160
Jul 31 16:23:31 infras kernel: [577035.822727]  do_page_fault+0x2c/0xe0
Jul 31 16:23:31 infras kernel: [577035.822730]  do_async_page_fault+0x39/0x70
Jul 31 16:23:31 infras kernel: [577035.822734]  async_page_fault+0x34/0x40
Jul 31 16:23:31 infras kernel: [577035.822736] RIP: 0033:0x563e3b75e760
Jul 31 16:23:31 infras kernel: [577035.822751] Code: Bad RIP value.
Jul 31 16:23:31 infras kernel: [577035.822753] RSP: 002b:00007fff391a0c78 EFLAGS: 00010202
Jul 31 16:23:31 infras kernel: [577035.822754] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000fffebff7
Jul 31 16:23:31 infras kernel: [577035.822755] RDX: 0000000000000046 RSI: 0000563e3b7dc4e9 RDI: 0000563e3d126170
Jul 31 16:23:31 infras kernel: [577035.822756] RBP: 0000563e3d126170 R08: 0000000000000005 R09: 0000563e3b7dc448
Jul 31 16:23:31 infras kernel: [577035.822757] R10: 0000563e3b7dc450 R11: 000000000000000e R12: 0000563e3d09b820
Jul 31 16:23:31 infras kernel: [577035.822758] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Jul 31 16:23:31 infras kernel: [577035.822760] Mem-Info:
Jul 31 16:23:31 infras kernel: [577035.822765] active_anon:1867136 inactive_anon:12010 isolated_anon:0
Jul 31 16:23:31 infras kernel: [577035.822765]  active_file:175 inactive_file:243 isolated_file:0
Jul 31 16:23:31 infras kernel: [577035.822765]  unevictable:4718 dirty:0 writeback:0 unstable:0
Jul 31 16:23:31 infras kernel: [577035.822765]  slab_reclaimable:16360 slab_unreclaimable:32930
Jul 31 16:23:31 infras kernel: [577035.822765]  mapped:41758 shmem:89452 pagetables:8500 bounce:0
Jul 31 16:23:31 infras kernel: [577035.822765]  free:25427 free_pcp:1192 free_cma:0
Jul 31 16:23:31 infras kernel: [577035.822769] Node 0 active_anon:7468544kB inactive_anon:48040kB active_file:700kB inactive_file:972kB unevictable:18872kB isolated(anon):0kB isolated(file):0kB mapped:166860kB dirty:0kB writeback:0kB shmem:357808kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Jul 31 16:23:31 infras kernel: [577035.822770] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822774] lowmem_reserve[]: 0 2916 7705 7705 7705
Jul 31 16:23:31 infras kernel: [577035.822776] Node 0 DMA32 free:43984kB min:25532kB low:31912kB high:38292kB active_anon:2859584kB inactive_anon:3500kB active_file:196kB inactive_file:224kB unevictable:36kB writepending:0kB present:3129200kB managed:3022280kB mlocked:36kB kernel_stack:2108kB pagetables:16292kB bounce:0kB free_pcp:2112kB local_pcp:180kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822780] lowmem_reserve[]: 0 0 4788 4788 4788
Jul 31 16:23:31 infras kernel: [577035.822783] Node 0 Normal free:41816kB min:41912kB low:52388kB high:62864kB active_anon:4608960kB inactive_anon:44540kB active_file:280kB inactive_file:920kB unevictable:18836kB writepending:0kB present:5046272kB managed:4903344kB mlocked:18836kB kernel_stack:15316kB pagetables:17708kB bounce:0kB free_pcp:2688kB local_pcp:680kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822787] lowmem_reserve[]: 0 0 0 0 0
Jul 31 16:23:31 infras kernel: [577035.822789] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 31 16:23:31 infras kernel: [577035.822810] Node 0 DMA32: 1091*4kB (UM) 1380*8kB (UME) 608*16kB (UME) 194*32kB (UME) 103*64kB (UME) 25*128kB (UME) 4*256kB (UME) 2*512kB (UM) 1*1024kB (E) 0*2048kB 0*4096kB = 44204kB
Jul 31 16:23:31 infras kernel: [577035.822831] Node 0 Normal: 938*4kB (UMEH) 530*8kB (UMEH) 244*16kB (UMEH) 184*32kB (UMEH) 45*64kB (UMEH) 11*128kB (UME) 5*256kB (UMH) 32*512kB (UMH) 3*1024kB (UM) 0*2048kB 0*4096kB = 42808kB
Jul 31 16:23:31 infras kernel: [577035.822841] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 31 16:23:31 infras kernel: [577035.822842] 91885 total pagecache pages
Jul 31 16:23:31 infras kernel: [577035.822843] 0 pages in swap cache
Jul 31 16:23:31 infras kernel: [577035.822844] Swap cache stats: add 0, delete 0, find 0/0
Jul 31 16:23:31 infras kernel: [577035.822845] Free swap  = 0kB
Jul 31 16:23:31 infras kernel: [577035.822845] Total swap = 0kB
Jul 31 16:23:31 infras kernel: [577035.822846] 2047866 pages RAM
Jul 31 16:23:31 infras kernel: [577035.822847] 0 pages HighMem/MovableOnly
Jul 31 16:23:31 infras kernel: [577035.822847] 62483 pages reserved
Jul 31 16:23:31 infras kernel: [577035.822848] 0 pages cma reserved
Jul 31 16:23:31 infras kernel: [577035.822848] 0 pages hwpoisoned
Jul 31 16:23:31 infras kernel: [577035.822849] Tasks state (memory values in pages):
Jul 31 16:23:31 infras kernel: [577035.822850] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul 31 16:23:31 infras kernel: [577035.822855] [    380]     0   380    21826      965   159744        0          -250 systemd-journal
Jul 31 16:23:31 infras kernel: [577035.822858] [    410]     0   410     5110      755    69632        0         -1000 systemd-udevd
Jul 31 16:23:31 infras kernel: [577035.822860] [    565]     0   565    70072     4524    86016        0         -1000 multipathd
Jul 31 16:23:31 infras kernel: [577035.822863] [    659]   100   659     6885      892    77824        0             0 systemd-network
Jul 31 16:23:31 infras kernel: [577035.822865] [    661]   101   661     6238     1237    81920        0             0 systemd-resolve
Jul 31 16:23:31 infras kernel: [577035.822867] [    697]     0   697    59432      543    94208        0             0 accounts-daemon
Jul 31 16:23:31 infras kernel: [577035.822869] [    703]     0   703     2142      528    53248        0             0 cron
Jul 31 16:23:31 infras kernel: [577035.822871] [    706]   103   706     1946      796    57344        0          -900 dbus-daemon
Jul 31 16:23:31 infras kernel: [577035.822873] [    719]     0   719    20473      746    53248        0             0 irqbalance
Jul 31 16:23:31 infras kernel: [577035.822875] [    723]   113   723   110413     8379   184320        0             0 memcached
Jul 31 16:23:31 infras kernel: [577035.822877] [    724]     0   724     7499     2861    94208        0             0 networkd-dispat
Jul 31 16:23:31 infras kernel: [577035.822879] [    727]     0   727   289789    11173   270336        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822881] [    728]     0   728   294791    11664   319488        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.822884] [    731]     0   731   303465    11716   323584        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822885] [    733]     0   733    58188      218    86016        0             0 polkitd
Jul 31 16:23:31 infras kernel: [577035.822887] [    737]     0   737     1611      586    49152        0             0 qemu-ga
Jul 31 16:23:31 infras kernel: [577035.822889] [    742]   104   742    56087      441    86016        0             0 rsyslogd
Jul 31 16:23:31 infras kernel: [577035.822892] [    748]     0   748   385484     3129   278528        0          -900 snapd
Jul 31 16:23:31 infras kernel: [577035.822894] [    758]     0   758     4441      999    77824        0             0 systemd-logind
Jul 31 16:23:31 infras kernel: [577035.822896] [    761]     0   761    98381      482   118784        0             0 udisksd
Jul 31 16:23:31 infras kernel: [577035.822898] [    765]     0   765      956      465    49152        0             0 atd
Jul 31 16:23:31 infras kernel: [577035.822900] [    767]     0   767    99245     2797   131072        0             0 f2b/server
Jul 31 16:23:31 infras kernel: [577035.822902] [    793]   117   793    18659      264    61440        0             0 ntpd
Jul 31 16:23:31 infras kernel: [577035.822904] [    794]     0   794     1838      436    53248        0             0 agetty
Jul 31 16:23:31 infras kernel: [577035.822906] [    812]     0   812     1457      361    45056        0             0 agetty
Jul 31 16:23:31 infras kernel: [577035.822908] [    835]     0   835    78785      920   110592        0             0 ModemManager
Jul 31 16:23:31 infras kernel: [577035.822910] [    843]     0   843     3054      809    61440        0         -1000 sshd
Jul 31 16:23:31 infras kernel: [577035.822913] [    887]     0   887    27101     2749   110592        0             0 unattended-upgr
Jul 31 16:23:31 infras kernel: [577035.822915] [    888]   116   888    24445    10574   184320        0             0 redis-server
Jul 31 16:23:31 infras kernel: [577035.822917] [   1346]    33  1346   289837     2272   200704        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822919] [   1349]    33  1349   289870     2289   200704        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822921] [   1695]    33  1695   304809    22150   438272        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822923] [   1736]    33  1736   308462    21937   471040        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822925] [   1763]    33  1763   304389    21593   434176        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822927] [   1912]    33  1912   304955    17318   442368        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822929] [   1919]    33  1919   305002    20665   446464        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822931] [   1951]    33  1951   304408    21288   442368        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822933] [   1952]    33  1952   305002    21823   450560        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822935] [   1954]    33  1954   308971    22513   479232        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822937] [   1956]    33  1956   308920    21774   466944        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822939] [   1964]    33  1964   305002    20881   446464        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822941] [   1971]    33  1971   305002    17471   446464        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822943] [   4509]   112  4509  1808848   318105  3014656        0             0 mariadbd
Jul 31 16:23:31 infras kernel: [577035.822945] [  16313]     0 16313     5311     2767    86016        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822947] [  17393]    33 17393   304430    21768   442368        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822949] [  17406]    33 17406   304942    22279   442368        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822951] [  17482]    33 17482   308442    18568   471040        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822953] [  17749]    33 17749      789      104    45056        0             0 defunct
Jul 31 16:23:31 infras kernel: [577035.822955] [  17750]    33 17750      790      123    45056        0             0 defunct
Jul 31 16:23:31 infras kernel: [577035.822957] [ 160484]    33 160484   308978    20822   479232        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822959] [ 160490]    33 160490   304475    17610   438272        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822961] [ 160491]    33 160491   304989    18461   442368        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822963] [ 160504]    33 160504   304939    16458   438272        0             0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822965] [ 608970]     0 608970     5098      752    65536        0             0 proftpd
Jul 31 16:23:31 infras kernel: [577035.822967] [1310851]    33 1310851     5245     2626    81920        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822969] [1310852]    33 1310852     5245     2335    81920        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822971] [1310853]    33 1310853   620479     4710   446464        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822973] [1310854]    33 1310854   620457     4670   442368        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822975] [1310855]    33 1310855   620412     3976   438272        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822977] [1310858]    33 1310858   620453     5046   446464        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822979] [1310872]    33 1310872   629012     5176   466944        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822981] [1310884]    33 1310884   620916     5330   450560        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822983] [1310898]    33 1310898   620453     4274   438272        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822985] [1310911]    33 1310911   620369     4844   438272        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822987] [1310922]    33 1310922   620474     3817   438272        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822989] [1310954]    33 1310954   620635     4973   442368        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822991] [1310967]    33 1310967   620527     4620   442368        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822993] [1310968]    33 1310968   620578     4872   446464        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822994] [1310969]    33 1310969   620618     4517   442368        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822996] [1311560]    33 1311560   621357     5954   450560        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.822998] [1311561]    33 1311561   620976     5337   446464        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.823000] [1311562]    33 1311562   620811     5355   442368        0             0 apache2
Jul 31 16:23:31 infras kernel: [577035.823002] [1340871]     0 1340871     3458     1028    65536        0             0 sshd
Jul 31 16:23:31 infras kernel: [577035.823005] [1340886]  1001 1340886     4859     1145    69632        0             0 systemd
Jul 31 16:23:31 infras kernel: [577035.823007] [1340887]  1001 1340887    43272     1233   106496        0             0 (sd-pam)
Jul 31 16:23:31 infras kernel: [577035.823009] [1341019]  1001 1341019     3493      625    65536        0             0 sshd
Jul 31 16:23:31 infras kernel: [577035.823011] [1341020]  1001 1341020     2513      942    49152        0             0 bash
Jul 31 16:23:31 infras kernel: [577035.823013] [1347748]    33 1347748   294908     3785   315392        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823015] [1347749]    33 1347749   294908     3837   311296        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823017] [1347752]    33 1347752   294908     4039   311296        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823020] [1347753]    33 1347753   294909     3977   311296        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823022] [1348570]    33 1348570   294909     3947   311296        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823024] [1348571]    33 1348571   294909     3616   311296        0             0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823026] [1348789]    33 1348789   290568    13864   339968        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823028] [1349147]    33 1349147   290574    13532   339968        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823030] [1349149]    33 1349149   290568    13445   335872        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823032] [1350813]    33 1350813   290571    13512   335872        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823034] [1352345]     0 1352345    11336     7259   126976        0             0 miniserv.pl
Jul 31 16:23:31 infras kernel: [577035.823036] [1352448]     0 1352448    10235     8448   122880        0             0 stats.pl
Jul 31 16:23:31 infras kernel: [577035.823039] [1352464]     0 1352464  1326549  1321860 10670080        0             0 /usr/share/webm
Jul 31 16:23:31 infras kernel: [577035.823041] [1352598]     0 1352598   289822     2175   184320        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823043] [1352599]     0 1352599   289789     1790   180224        0             0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823044] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/webmin.service,task=/usr/share/webm,pid=1352464,uid=0
Jul 31 16:23:31 infras kernel: [577035.823067] Out of memory: Killed process 1352464 (/usr/share/webm) total-vm:5306196kB, anon-rss:5286396kB, file-rss:1044kB, shmem-rss:0kB, UID:0 pgtables:10420kB oom_score_adj:0

[iliajie]

Nothing can resolve it

What have you tied?

Can you try installing the latest nightly version of Webmin to check if that resolves the problem for you?

After installation is finished force restart Webmin by running /etc/webmin/restart-by-force-kill command.

Jul 31 16:23:31 infras kernel: [577035.823067] Out of memory: Killed process 1352464 (/usr/share/webm) total-vm:5306196kB, anon-rss:5286396kB, file-rss:1044kB, shmem-rss:0kB, UID:0 pgtables:10420kB oom_score_adj:0

5.04 GB is way to much for Webmin process. I have never seen it happen.

[SylvainLegrand]

I tied the lines from my syslog before the crash.

the nightly version with /etc/webmin/restart-by-force-kill command doesn't change anything !

[iliajie]

the nightly version with /etc/webmin/restart-by-force-kill command doesn't change anything !

What is the output of the following commands on your system:

free -h

grep websock /etc/webmin/miniserv.conf

head -n 1 /usr/share/webmin/authentic-theme/stats.pl

tail -n 20 /var/webmin/miniserv.error

find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}'

[SylvainLegrand]

free -h Mem: 7.6Gi 796Mi 6.4Gi 125Mi 427Mi 6.4Gi Swap: 8.0Gi 1.1Gi 6.9Gi

sudo grep websock /etc/webmin/miniserv.conf websockets_/authentic-theme/ws-556=host=127.0.0.1 port=556 wspath=/ user=infras time=1722436870 websockets_/authentic-theme/ws-555=host=127.0.0.1 port=555 wspath=/ user=infras time=1722457578

sudo head -n 1 /usr/share/webmin/authentic-theme/stats.pl #!/usr/bin/perl

sudo tail -n 20 /var/webmin/miniserv.error `[31/Jul/2024:16:41:05 +0200] PAM authentication enabled [31/Jul/2024:16:41:10 +0200] Reloading configuration [31/Jul/2024:16:43:07 +0200] [2001:41d0:305:2100::a0ca] /extensions/file-manager/tree.cgi : Perl execution failed : Can't cd to ../../.. from /proc/4944/task/4944/ns: No such process at /usr/share/webmin/vendor_perl/File/Find.pm line 469.

[31/Jul/2024:17:17:58 +0200] Reloading configuration [31/Jul/2024:22:24:08 +0200] Reloading configuration [31/Jul/2024:22:25:44 +0200] Restarting [31/Jul/2024:22:25:47 +0200] miniserv.pl started [31/Jul/2024:22:25:47 +0200] IPv6 support enabled [31/Jul/2024:22:25:47 +0200] Using MD5 module Digest::MD5 [31/Jul/2024:22:25:47 +0200] Using SHA512 module Crypt::SHA [31/Jul/2024:22:25:47 +0200] PAM authentication enabled [31/Jul/2024:22:25:59 +0200] Shutting down [31/Jul/2024:22:25:59 +0200] Shutting down [31/Jul/2024:22:26:02 +0200] miniserv.pl started [31/Jul/2024:22:26:02 +0200] IPv6 support enabled [31/Jul/2024:22:26:02 +0200] Using MD5 module Digest::MD5 [31/Jul/2024:22:26:02 +0200] Using SHA512 module Crypt::SHA [31/Jul/2024:22:26:02 +0200] PAM authentication enabled [31/Jul/2024:22:26:18 +0200] Reloading configuration`

sudo find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}' Filename: /var/webmin/modules/authentic-theme/stats-server-555.log cat: /var/webmin/modules/authentic-theme/stats-server-555.log: Permission denied Filename: /var/webmin/modules/authentic-theme/stats-server-556.log cat: /var/webmin/modules/authentic-theme/stats-server-556.log: Permission denied

[iliajie]

sudo find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}' Filename: /var/webmin/modules/authentic-theme/stats-server-555.log cat: /var/webmin/modules/authentic-theme/stats-server-555.log: Permission denied Filename: /var/webmin/modules/authentic-theme/stats-server-556.log cat: /var/webmin/modules/authentic-theme/stats-server-556.log: Permission denied

Try this instead:

sudo sh -c 'find /var/webmin/modules/authentic-theme/ -name "stats-server*" -print | xargs -I {} sh -c "echo Filename: {}; cat {}"'

websockets_/authentic-theme/ws-555=host=127.0.0.1 port=555 wspath=/ user=infras time=1722457578

Also, can you try logging in as root rather than infras user and see if it changes anything for you?

[SylvainLegrand]

Filename: /var/webmin/modules/authentic-theme/stats-server-555.log
[31/Jul/2024:22:26:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 555
[31/Jul/2024:22:26:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40978 opened
[31/Jul/2024:22:26:28 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40978 closed
[31/Jul/2024:22:26:33 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40984 opened
[31/Jul/2024:22:26:35 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40984 closed
[31/Jul/2024:22:26:39 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33354 opened
[31/Jul/2024:22:26:41 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33354 closed
[31/Jul/2024:22:26:43 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33366 opened
[31/Jul/2024:22:26:45 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33366 closed
[31/Jul/2024:22:26:49 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 opened
[31/Jul/2024:22:26:51 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 closed
[31/Jul/2024:22:27:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 is closed due to inactivity
[31/Jul/2024:22:38:23 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 41370 opened
[31/Jul/2024:22:38:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 41370 closed
[31/Jul/2024:22:38:28 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39874 opened
[31/Jul/2024:22:38:30 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39874 closed
[31/Jul/2024:22:38:34 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39880 opened
[31/Jul/2024:22:38:36 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39880 closed
[31/Jul/2024:22:38:40 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 60156 opened
[31/Jul/2024:22:38:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 60156 closed
[31/Jul/2024:22:38:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 opened
[31/Jul/2024:22:38:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 closed
[31/Jul/2024:22:39:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 is closed due to inactivity
Filename: /var/webmin/modules/authentic-theme/stats-server-556.log
[31/Jul/2024:16:41:11 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 556
[31/Jul/2024:16:41:15 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 55968 opened
[31/Jul/2024:16:41:17 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 55968 closed
[31/Jul/2024:16:41:17 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39326 opened
[31/Jul/2024:16:41:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39326 closed
[31/Jul/2024:16:41:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39332 opened
[31/Jul/2024:16:41:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39332 closed
[31/Jul/2024:16:41:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 opened
[31/Jul/2024:16:41:23 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 closed
[31/Jul/2024:16:41:36 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 is closed due to inactivity
[31/Jul/2024:17:17:59 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 opened
[31/Jul/2024:17:18:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 closed
[31/Jul/2024:17:18:14 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 is closed due to inactivity
[31/Jul/2024:18:18:00 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 opened
[31/Jul/2024:18:18:02 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 closed
[31/Jul/2024:18:18:15 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 is closed due to inactivity
[31/Jul/2024:19:18:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 opened
[31/Jul/2024:19:18:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 closed
[31/Jul/2024:19:18:16 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 is closed due to inactivity
[31/Jul/2024:20:18:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 opened
[31/Jul/2024:20:18:05 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 closed
[31/Jul/2024:20:18:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 is closed due to inactivity
[31/Jul/2024:21:18:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 opened
[31/Jul/2024:21:18:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 closed
[31/Jul/2024:21:18:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 is closed due to inactivity
[31/Jul/2024:22:18:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 opened
[31/Jul/2024:22:18:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 closed
[31/Jul/2024:22:18:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 is closed due to inactivity
[31/Jul/2024:22:21:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57974 opened
[31/Jul/2024:22:21:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57974 closed
[31/Jul/2024:22:21:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57988 opened
[31/Jul/2024:22:21:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57988 closed
[31/Jul/2024:22:21:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57994 opened
[31/Jul/2024:22:21:10 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57994 closed
[31/Jul/2024:22:21:10 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 58010 opened
[31/Jul/2024:22:21:12 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 58010 closed
[31/Jul/2024:22:21:16 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 opened
[31/Jul/2024:22:21:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 closed
[31/Jul/2024:22:21:31 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 is closed due to inactivity