Closed UweLoyal closed 1 month ago
I found a similar problem : I use webmin on 5 servers. For 4 of them the new version works properly.
On the last one the real time monitoring page crash with an out of memory !
If I refresh the page the memory consumed by webmin increase until the crash (I use htop) the command is "/usr/share/webmin/authentic-theme/sysinfo.cgi"
Greetings, Sylvain
@iliajie please take a look at this ASAP
For 4 of them the new version works properly.
On the last one the real time monitoring page crash with an out of memory !
I wonder what makes those servers different? Can you provide more debug information? Is there anything useful logged under /var/webmin
and /var/webmin/modules/authentic-theme
? Also what is logged in browser's console for that server that doesn't work?
If I refresh the page the memory consumed by webmin increase until the crash (I use htop)
I have run numerous tests and never had this issue. Can you please tell more about the system which is failing? How is it different in terms of configuration?
I found something very strange :
On /var/webmin/modules/authentic-theme
the files stats-server-556.log
, stats-server-557.log
, stats-server-558.log
, stats-server-559.log
, and stats-server-560.log
contain only one line : "Failed to run /usr/share/webmin/authentic-theme/stats.pl : No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."
But the file /usr/share/webmin/authentic-theme/stats.pl
exist with chmod = 755 and chown = root:root
Also I use a reverse proxy to connect to webmin. But is like my other servers.
No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."
This is an odd one! Do you have /etc/webmin/authentic-theme
directory?
yes
No such file or directory at /etc/webmin/authentic-theme/stats.pl line 13."
This is an odd one! Do you have
/etc/webmin/authentic-theme
directory?
Oh, I think I see. It's the PERLLIB
or $root
isn't set correctly.
What is the output of:
grep -iE 'PERLLIB|root' /etc/webmin/config /etc/webmin/miniserv.conf
/etc/webmin/miniserv.conf:root=/usr/share/webmin /etc/webmin/miniserv.conf:preroot=authentic-theme
/etc/webmin/authentic-theme/stats.pl
That looks fine!
What is the content of /etc/webmin/authentic-theme/stats.pl
file?
#!/usr/bin/perl
open(CONF, "</etc/webmin/miniserv.conf") || die "Failed to open /etc/webmin/miniserv.conf : $!";
while(<CONF>) {
$root = $1 if (/^root=(.*)/);
}
close(CONF);
$root || die "No root= line found in /etc/webmin/miniserv.conf";
$ENV{'PERLLIB'} = "$root";
$ENV{'WEBMIN_CONFIG'} = "/etc/webmin";
$ENV{'WEBMIN_VAR'} = "/var/webmin";
delete($ENV{'MINISERV_CONFIG'});
chdir("$root/authentic-theme");
exec("$root/authentic-theme/stats.pl", @ARGV) || die "Failed to run $root/authentic-theme/stats.pl : $!";
That's odd. I don't understand the problem.
Is this Webmin instance running via a proxy? If so, do you experience the same issue when connected directly?
Does /usr/bin/perl exist on your system?
Also, does /usr/share/webmin/authentic-theme/stats.pl exist and what's the first line of that file?
/usr/bin/perl
does exist
/usr/share/webmin/authentic-theme/stats.pl
does exist and the first line is #!/usr/local/bin/perl
/usr/local/bin/perl
doesn't exist !!!!!
and the first line is
#!/usr/local/bin/perl
Oh, no! But why? Jamie, Webmin should replace it, right?
Oh shit there's the problem ... let me see what's up with that
How was Webmin installed on this system exactly?
Some years ago I do : `sudo wget https://download.webmin.com/jcameron-key.asc
sudo apt-key add jcameron-key.asc
sudo add-apt-repository 'deb http://download.webmin.com/download/repository sarge contrib'
sudo apt-get update
sudo apt-get install webmin -y ` Until now the updates works properly.
But we do replace /usr/local/bin/perl
with /usr/bin/perl
.
What is the output of:
ls -lsa /usr/local/bin/perl
ls: cannot access '/usr/local/bin/perl': No such file or directory
Do any other files in /usr/share/webmin/authentic-theme
contain /usr/local/bin/perl
?
Does it fix the problem if you:
ln -s /usr/bin/perl /usr/local/bin/perl
Do any other files in
/usr/share/webmin/authentic-theme
contain/usr/local/bin/perl
?
Or any other Webmin modules, for example acl/
?
all the files on /usr/share/webmin/authentic-theme
contain /usr/local/bin/perl
How about other dirs like /usr/share/webmin/acl
?
all the files on
/usr/share/webmin/authentic-theme
contain/usr/local/bin/perl
Did you upgraded theme from the repo, using Theme Configuration page?
yes, of course !
And you did this after Webmin 2.201 was installed?
files on acl folder only contain #!/usr/bin/perl
And you did this after Webmin 2.201 was installed? YES
And you did this after Webmin 2.201 was installed? YES
That explains it! I will address it! This isn't hard to do! And we don't need a new Webmin release, which is a relief ..
Does it fix the problem if you:
ln -s /usr/bin/perl /usr/local/bin/perl
You can work around it by doing this.
Does it fix the problem if you:
ln -s /usr/bin/perl /usr/local/bin/perl
You can work around it by doing this.
It doesn't work !
Best thing would be to re-install Webmin and over-write any changed files with apt-get reinstall webmin
This issue with shebang was addressed in this patch https://github.com/webmin/authentic-theme/commit/24464c9dbfc3122d25334703927ef4b15fa7df80.
Ok. I re-installed webmin :
on authentic-theme files, now the right path is writing
The log on webmin/modules/authentic-theme is now stats-server-555.log
It's content :
[25/Jul/2024:00:22:14 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 555
[25/Jul/2024:00:22:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 50640 opened
[25/Jul/2024:00:22:20 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 50640 closed
[25/Jul/2024:00:22:20 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56568 opened
[25/Jul/2024:00:22:22 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56568 closed
[25/Jul/2024:00:22:22 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56578 opened
[25/Jul/2024:00:22:24 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56578 closed
[25/Jul/2024:00:22:24 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 opened
[25/Jul/2024:00:22:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 closed
[25/Jul/2024:00:22:39 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56586 is closed due to inactivity
[25/Jul/2024:00:34:40 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39210 opened
[25/Jul/2024:00:34:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39210 closed
[25/Jul/2024:00:34:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39214 opened
[25/Jul/2024:00:34:44 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39214 closed
[25/Jul/2024:00:34:44 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39226 opened
[25/Jul/2024:00:34:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39226 closed
[25/Jul/2024:00:34:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39234 opened
[25/Jul/2024:00:34:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39234 closed
[25/Jul/2024:00:34:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 opened
[25/Jul/2024:00:34:50 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 closed
[25/Jul/2024:00:35:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39242 is closed due to inactivity
But I face the same problem !
the command /usr/share/webmin/authentic-theme/sysinfo.cgi
crash with an out of memory !
And a second command crash /usr/share/webmin/authentic-theme/xhr.cgi
The result !
If I disable the reverse proxy I can connect thru the IP address (custom port - not 10000) but webmin behavior is the same ! No stats page !
What if you force reload Webmin by running:
/etc/webmin/restart-by-force-kill
Still the same problem
This issue with shebang was addressed in this patch webmin/authentic-theme@24464c9.
I had exactly the sam problem after upgrade to latest webmin version (websocket connection error) and downgrading authentic-theme by running following commands solved issue for me.
cd /usr/share/webmin/authentic-theme
./theme-update.sh --release:21.10 --force
Very good Idea, now also works fine on my System!
stats.cgi call looks difference!
If I disable the reverse proxy I can connect thru the IP address (custom port - not 10000) but webmin behavior is the same !
I really wish I could help, but sadly, I cannot reproduce this issue and can't understand what's causing it on your end either.
Perhaps, for now, the right solution would be to disable live monitoring on the Theme Configuration page rather than downgrading the theme.
Nothing can resolve it !!!!
This is my syslog about the memory crash
Jul 31 16:23:31 infras kernel: [577035.822644] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jul 31 16:23:31 infras kernel: [577035.822648] CPU: 3 PID: 1 Comm: systemd Not tainted 5.4.0-189-generic #209-Ubuntu
Jul 31 16:23:31 infras kernel: [577035.822649] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 2:1.10.2-58953eb7 04/01/2014
Jul 31 16:23:31 infras kernel: [577035.822650] Call Trace:
Jul 31 16:23:31 infras kernel: [577035.822658] dump_stack+0x6d/0x8b
Jul 31 16:23:31 infras kernel: [577035.822662] dump_header+0x4f/0x1eb
Jul 31 16:23:31 infras kernel: [577035.822664] oom_kill_process.cold+0xb/0x10
Jul 31 16:23:31 infras kernel: [577035.822668] out_of_memory+0x1cf/0x500
Jul 31 16:23:31 infras kernel: [577035.822672] __alloc_pages_slowpath+0xe14/0xee0
Jul 31 16:23:31 infras kernel: [577035.822675] __alloc_pages_nodemask+0x2d0/0x320
Jul 31 16:23:31 infras kernel: [577035.822678] alloc_pages_current+0x87/0xe0
Jul 31 16:23:31 infras kernel: [577035.822682] __page_cache_alloc+0x72/0x90
Jul 31 16:23:31 infras kernel: [577035.822685] pagecache_get_page+0xbf/0x300
Jul 31 16:23:31 infras kernel: [577035.822687] filemap_fault+0x79e/0xbc0
Jul 31 16:23:31 infras kernel: [577035.822690] ? unlock_page_memcg+0x12/0x20
Jul 31 16:23:31 infras kernel: [577035.822693] ? page_add_file_rmap+0xff/0x1a0
Jul 31 16:23:31 infras kernel: [577035.822696] ? xas_load+0xd/0x80
Jul 31 16:23:31 infras kernel: [577035.822699] ? xas_find+0x17f/0x1c0
Jul 31 16:23:31 infras kernel: [577035.822701] ? filemap_map_pages+0x24c/0x380
Jul 31 16:23:31 infras kernel: [577035.822705] ext4_filemap_fault+0x32/0x50
Jul 31 16:23:31 infras kernel: [577035.822708] __do_fault+0x3c/0x170
Jul 31 16:23:31 infras kernel: [577035.822710] do_fault+0x24b/0x640
Jul 31 16:23:31 infras kernel: [577035.822712] __handle_mm_fault+0x4c5/0x7a0
Jul 31 16:23:31 infras kernel: [577035.822715] handle_mm_fault+0xca/0x200
Jul 31 16:23:31 infras kernel: [577035.822719] do_user_addr_fault+0x1f9/0x450
Jul 31 16:23:31 infras kernel: [577035.822721] __do_page_fault+0x58/0x90
Jul 31 16:23:31 infras kernel: [577035.822725] ? exit_to_usermode_loop+0xea/0x160
Jul 31 16:23:31 infras kernel: [577035.822727] do_page_fault+0x2c/0xe0
Jul 31 16:23:31 infras kernel: [577035.822730] do_async_page_fault+0x39/0x70
Jul 31 16:23:31 infras kernel: [577035.822734] async_page_fault+0x34/0x40
Jul 31 16:23:31 infras kernel: [577035.822736] RIP: 0033:0x563e3b75e760
Jul 31 16:23:31 infras kernel: [577035.822751] Code: Bad RIP value.
Jul 31 16:23:31 infras kernel: [577035.822753] RSP: 002b:00007fff391a0c78 EFLAGS: 00010202
Jul 31 16:23:31 infras kernel: [577035.822754] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000fffebff7
Jul 31 16:23:31 infras kernel: [577035.822755] RDX: 0000000000000046 RSI: 0000563e3b7dc4e9 RDI: 0000563e3d126170
Jul 31 16:23:31 infras kernel: [577035.822756] RBP: 0000563e3d126170 R08: 0000000000000005 R09: 0000563e3b7dc448
Jul 31 16:23:31 infras kernel: [577035.822757] R10: 0000563e3b7dc450 R11: 000000000000000e R12: 0000563e3d09b820
Jul 31 16:23:31 infras kernel: [577035.822758] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Jul 31 16:23:31 infras kernel: [577035.822760] Mem-Info:
Jul 31 16:23:31 infras kernel: [577035.822765] active_anon:1867136 inactive_anon:12010 isolated_anon:0
Jul 31 16:23:31 infras kernel: [577035.822765] active_file:175 inactive_file:243 isolated_file:0
Jul 31 16:23:31 infras kernel: [577035.822765] unevictable:4718 dirty:0 writeback:0 unstable:0
Jul 31 16:23:31 infras kernel: [577035.822765] slab_reclaimable:16360 slab_unreclaimable:32930
Jul 31 16:23:31 infras kernel: [577035.822765] mapped:41758 shmem:89452 pagetables:8500 bounce:0
Jul 31 16:23:31 infras kernel: [577035.822765] free:25427 free_pcp:1192 free_cma:0
Jul 31 16:23:31 infras kernel: [577035.822769] Node 0 active_anon:7468544kB inactive_anon:48040kB active_file:700kB inactive_file:972kB unevictable:18872kB isolated(anon):0kB isolated(file):0kB mapped:166860kB dirty:0kB writeback:0kB shmem:357808kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Jul 31 16:23:31 infras kernel: [577035.822770] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822774] lowmem_reserve[]: 0 2916 7705 7705 7705
Jul 31 16:23:31 infras kernel: [577035.822776] Node 0 DMA32 free:43984kB min:25532kB low:31912kB high:38292kB active_anon:2859584kB inactive_anon:3500kB active_file:196kB inactive_file:224kB unevictable:36kB writepending:0kB present:3129200kB managed:3022280kB mlocked:36kB kernel_stack:2108kB pagetables:16292kB bounce:0kB free_pcp:2112kB local_pcp:180kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822780] lowmem_reserve[]: 0 0 4788 4788 4788
Jul 31 16:23:31 infras kernel: [577035.822783] Node 0 Normal free:41816kB min:41912kB low:52388kB high:62864kB active_anon:4608960kB inactive_anon:44540kB active_file:280kB inactive_file:920kB unevictable:18836kB writepending:0kB present:5046272kB managed:4903344kB mlocked:18836kB kernel_stack:15316kB pagetables:17708kB bounce:0kB free_pcp:2688kB local_pcp:680kB free_cma:0kB
Jul 31 16:23:31 infras kernel: [577035.822787] lowmem_reserve[]: 0 0 0 0 0
Jul 31 16:23:31 infras kernel: [577035.822789] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
Jul 31 16:23:31 infras kernel: [577035.822810] Node 0 DMA32: 1091*4kB (UM) 1380*8kB (UME) 608*16kB (UME) 194*32kB (UME) 103*64kB (UME) 25*128kB (UME) 4*256kB (UME) 2*512kB (UM) 1*1024kB (E) 0*2048kB 0*4096kB = 44204kB
Jul 31 16:23:31 infras kernel: [577035.822831] Node 0 Normal: 938*4kB (UMEH) 530*8kB (UMEH) 244*16kB (UMEH) 184*32kB (UMEH) 45*64kB (UMEH) 11*128kB (UME) 5*256kB (UMH) 32*512kB (UMH) 3*1024kB (UM) 0*2048kB 0*4096kB = 42808kB
Jul 31 16:23:31 infras kernel: [577035.822841] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 31 16:23:31 infras kernel: [577035.822842] 91885 total pagecache pages
Jul 31 16:23:31 infras kernel: [577035.822843] 0 pages in swap cache
Jul 31 16:23:31 infras kernel: [577035.822844] Swap cache stats: add 0, delete 0, find 0/0
Jul 31 16:23:31 infras kernel: [577035.822845] Free swap = 0kB
Jul 31 16:23:31 infras kernel: [577035.822845] Total swap = 0kB
Jul 31 16:23:31 infras kernel: [577035.822846] 2047866 pages RAM
Jul 31 16:23:31 infras kernel: [577035.822847] 0 pages HighMem/MovableOnly
Jul 31 16:23:31 infras kernel: [577035.822847] 62483 pages reserved
Jul 31 16:23:31 infras kernel: [577035.822848] 0 pages cma reserved
Jul 31 16:23:31 infras kernel: [577035.822848] 0 pages hwpoisoned
Jul 31 16:23:31 infras kernel: [577035.822849] Tasks state (memory values in pages):
Jul 31 16:23:31 infras kernel: [577035.822850] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jul 31 16:23:31 infras kernel: [577035.822855] [ 380] 0 380 21826 965 159744 0 -250 systemd-journal
Jul 31 16:23:31 infras kernel: [577035.822858] [ 410] 0 410 5110 755 69632 0 -1000 systemd-udevd
Jul 31 16:23:31 infras kernel: [577035.822860] [ 565] 0 565 70072 4524 86016 0 -1000 multipathd
Jul 31 16:23:31 infras kernel: [577035.822863] [ 659] 100 659 6885 892 77824 0 0 systemd-network
Jul 31 16:23:31 infras kernel: [577035.822865] [ 661] 101 661 6238 1237 81920 0 0 systemd-resolve
Jul 31 16:23:31 infras kernel: [577035.822867] [ 697] 0 697 59432 543 94208 0 0 accounts-daemon
Jul 31 16:23:31 infras kernel: [577035.822869] [ 703] 0 703 2142 528 53248 0 0 cron
Jul 31 16:23:31 infras kernel: [577035.822871] [ 706] 103 706 1946 796 57344 0 -900 dbus-daemon
Jul 31 16:23:31 infras kernel: [577035.822873] [ 719] 0 719 20473 746 53248 0 0 irqbalance
Jul 31 16:23:31 infras kernel: [577035.822875] [ 723] 113 723 110413 8379 184320 0 0 memcached
Jul 31 16:23:31 infras kernel: [577035.822877] [ 724] 0 724 7499 2861 94208 0 0 networkd-dispat
Jul 31 16:23:31 infras kernel: [577035.822879] [ 727] 0 727 289789 11173 270336 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822881] [ 728] 0 728 294791 11664 319488 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.822884] [ 731] 0 731 303465 11716 323584 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822885] [ 733] 0 733 58188 218 86016 0 0 polkitd
Jul 31 16:23:31 infras kernel: [577035.822887] [ 737] 0 737 1611 586 49152 0 0 qemu-ga
Jul 31 16:23:31 infras kernel: [577035.822889] [ 742] 104 742 56087 441 86016 0 0 rsyslogd
Jul 31 16:23:31 infras kernel: [577035.822892] [ 748] 0 748 385484 3129 278528 0 -900 snapd
Jul 31 16:23:31 infras kernel: [577035.822894] [ 758] 0 758 4441 999 77824 0 0 systemd-logind
Jul 31 16:23:31 infras kernel: [577035.822896] [ 761] 0 761 98381 482 118784 0 0 udisksd
Jul 31 16:23:31 infras kernel: [577035.822898] [ 765] 0 765 956 465 49152 0 0 atd
Jul 31 16:23:31 infras kernel: [577035.822900] [ 767] 0 767 99245 2797 131072 0 0 f2b/server
Jul 31 16:23:31 infras kernel: [577035.822902] [ 793] 117 793 18659 264 61440 0 0 ntpd
Jul 31 16:23:31 infras kernel: [577035.822904] [ 794] 0 794 1838 436 53248 0 0 agetty
Jul 31 16:23:31 infras kernel: [577035.822906] [ 812] 0 812 1457 361 45056 0 0 agetty
Jul 31 16:23:31 infras kernel: [577035.822908] [ 835] 0 835 78785 920 110592 0 0 ModemManager
Jul 31 16:23:31 infras kernel: [577035.822910] [ 843] 0 843 3054 809 61440 0 -1000 sshd
Jul 31 16:23:31 infras kernel: [577035.822913] [ 887] 0 887 27101 2749 110592 0 0 unattended-upgr
Jul 31 16:23:31 infras kernel: [577035.822915] [ 888] 116 888 24445 10574 184320 0 0 redis-server
Jul 31 16:23:31 infras kernel: [577035.822917] [ 1346] 33 1346 289837 2272 200704 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822919] [ 1349] 33 1349 289870 2289 200704 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.822921] [ 1695] 33 1695 304809 22150 438272 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822923] [ 1736] 33 1736 308462 21937 471040 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822925] [ 1763] 33 1763 304389 21593 434176 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822927] [ 1912] 33 1912 304955 17318 442368 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822929] [ 1919] 33 1919 305002 20665 446464 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822931] [ 1951] 33 1951 304408 21288 442368 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822933] [ 1952] 33 1952 305002 21823 450560 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822935] [ 1954] 33 1954 308971 22513 479232 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822937] [ 1956] 33 1956 308920 21774 466944 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822939] [ 1964] 33 1964 305002 20881 446464 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822941] [ 1971] 33 1971 305002 17471 446464 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822943] [ 4509] 112 4509 1808848 318105 3014656 0 0 mariadbd
Jul 31 16:23:31 infras kernel: [577035.822945] [ 16313] 0 16313 5311 2767 86016 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822947] [ 17393] 33 17393 304430 21768 442368 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822949] [ 17406] 33 17406 304942 22279 442368 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822951] [ 17482] 33 17482 308442 18568 471040 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822953] [ 17749] 33 17749 789 104 45056 0 0 defunct
Jul 31 16:23:31 infras kernel: [577035.822955] [ 17750] 33 17750 790 123 45056 0 0 defunct
Jul 31 16:23:31 infras kernel: [577035.822957] [ 160484] 33 160484 308978 20822 479232 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822959] [ 160490] 33 160490 304475 17610 438272 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822961] [ 160491] 33 160491 304989 18461 442368 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822963] [ 160504] 33 160504 304939 16458 438272 0 0 php-fpm8.3
Jul 31 16:23:31 infras kernel: [577035.822965] [ 608970] 0 608970 5098 752 65536 0 0 proftpd
Jul 31 16:23:31 infras kernel: [577035.822967] [1310851] 33 1310851 5245 2626 81920 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822969] [1310852] 33 1310852 5245 2335 81920 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822971] [1310853] 33 1310853 620479 4710 446464 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822973] [1310854] 33 1310854 620457 4670 442368 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822975] [1310855] 33 1310855 620412 3976 438272 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822977] [1310858] 33 1310858 620453 5046 446464 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822979] [1310872] 33 1310872 629012 5176 466944 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822981] [1310884] 33 1310884 620916 5330 450560 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822983] [1310898] 33 1310898 620453 4274 438272 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822985] [1310911] 33 1310911 620369 4844 438272 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822987] [1310922] 33 1310922 620474 3817 438272 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822989] [1310954] 33 1310954 620635 4973 442368 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822991] [1310967] 33 1310967 620527 4620 442368 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822993] [1310968] 33 1310968 620578 4872 446464 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822994] [1310969] 33 1310969 620618 4517 442368 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822996] [1311560] 33 1311560 621357 5954 450560 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.822998] [1311561] 33 1311561 620976 5337 446464 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.823000] [1311562] 33 1311562 620811 5355 442368 0 0 apache2
Jul 31 16:23:31 infras kernel: [577035.823002] [1340871] 0 1340871 3458 1028 65536 0 0 sshd
Jul 31 16:23:31 infras kernel: [577035.823005] [1340886] 1001 1340886 4859 1145 69632 0 0 systemd
Jul 31 16:23:31 infras kernel: [577035.823007] [1340887] 1001 1340887 43272 1233 106496 0 0 (sd-pam)
Jul 31 16:23:31 infras kernel: [577035.823009] [1341019] 1001 1341019 3493 625 65536 0 0 sshd
Jul 31 16:23:31 infras kernel: [577035.823011] [1341020] 1001 1341020 2513 942 49152 0 0 bash
Jul 31 16:23:31 infras kernel: [577035.823013] [1347748] 33 1347748 294908 3785 315392 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823015] [1347749] 33 1347749 294908 3837 311296 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823017] [1347752] 33 1347752 294908 4039 311296 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823020] [1347753] 33 1347753 294909 3977 311296 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823022] [1348570] 33 1348570 294909 3947 311296 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823024] [1348571] 33 1348571 294909 3616 311296 0 0 php-fpm8.1
Jul 31 16:23:31 infras kernel: [577035.823026] [1348789] 33 1348789 290568 13864 339968 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823028] [1349147] 33 1349147 290574 13532 339968 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823030] [1349149] 33 1349149 290568 13445 335872 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823032] [1350813] 33 1350813 290571 13512 335872 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823034] [1352345] 0 1352345 11336 7259 126976 0 0 miniserv.pl
Jul 31 16:23:31 infras kernel: [577035.823036] [1352448] 0 1352448 10235 8448 122880 0 0 stats.pl
Jul 31 16:23:31 infras kernel: [577035.823039] [1352464] 0 1352464 1326549 1321860 10670080 0 0 /usr/share/webm
Jul 31 16:23:31 infras kernel: [577035.823041] [1352598] 0 1352598 289822 2175 184320 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823043] [1352599] 0 1352599 289789 1790 180224 0 0 php-fpm7.4
Jul 31 16:23:31 infras kernel: [577035.823044] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/webmin.service,task=/usr/share/webm,pid=1352464,uid=0
Jul 31 16:23:31 infras kernel: [577035.823067] Out of memory: Killed process 1352464 (/usr/share/webm) total-vm:5306196kB, anon-rss:5286396kB, file-rss:1044kB, shmem-rss:0kB, UID:0 pgtables:10420kB oom_score_adj:0
Nothing can resolve it
What have you tied?
Can you try installing the latest nightly version of Webmin to check if that resolves the problem for you?
After installation is finished force restart Webmin by running /etc/webmin/restart-by-force-kill
command.
Jul 31 16:23:31 infras kernel: [577035.823067] Out of memory: Killed process 1352464 (/usr/share/webm) total-vm:5306196kB, anon-rss:5286396kB, file-rss:1044kB, shmem-rss:0kB, UID:0 pgtables:10420kB oom_score_adj:0
5.04 GB is way to much for Webmin process. I have never seen it happen.
I tied the lines from my syslog before the crash.
the nightly version with /etc/webmin/restart-by-force-kill command doesn't change anything !
the nightly version with /etc/webmin/restart-by-force-kill command doesn't change anything !
What is the output of the following commands on your system:
free -h
grep websock /etc/webmin/miniserv.conf
head -n 1 /usr/share/webmin/authentic-theme/stats.pl
tail -n 20 /var/webmin/miniserv.error
find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}'
free -h
Mem: 7.6Gi 796Mi 6.4Gi 125Mi 427Mi 6.4Gi Swap: 8.0Gi 1.1Gi 6.9Gi
sudo grep websock /etc/webmin/miniserv.conf
websockets_/authentic-theme/ws-556=host=127.0.0.1 port=556 wspath=/ user=infras time=1722436870 websockets_/authentic-theme/ws-555=host=127.0.0.1 port=555 wspath=/ user=infras time=1722457578
sudo head -n 1 /usr/share/webmin/authentic-theme/stats.pl
#!/usr/bin/perl
sudo tail -n 20 /var/webmin/miniserv.error `[31/Jul/2024:16:41:05 +0200] PAM authentication enabled [31/Jul/2024:16:41:10 +0200] Reloading configuration [31/Jul/2024:16:43:07 +0200] [2001:41d0:305:2100::a0ca] /extensions/file-manager/tree.cgi : Perl execution failed : Can't cd to ../../.. from /proc/4944/task/4944/ns: No such process at /usr/share/webmin/vendor_perl/File/Find.pm line 469.
[31/Jul/2024:17:17:58 +0200] Reloading configuration [31/Jul/2024:22:24:08 +0200] Reloading configuration [31/Jul/2024:22:25:44 +0200] Restarting [31/Jul/2024:22:25:47 +0200] miniserv.pl started [31/Jul/2024:22:25:47 +0200] IPv6 support enabled [31/Jul/2024:22:25:47 +0200] Using MD5 module Digest::MD5 [31/Jul/2024:22:25:47 +0200] Using SHA512 module Crypt::SHA [31/Jul/2024:22:25:47 +0200] PAM authentication enabled [31/Jul/2024:22:25:59 +0200] Shutting down [31/Jul/2024:22:25:59 +0200] Shutting down [31/Jul/2024:22:26:02 +0200] miniserv.pl started [31/Jul/2024:22:26:02 +0200] IPv6 support enabled [31/Jul/2024:22:26:02 +0200] Using MD5 module Digest::MD5 [31/Jul/2024:22:26:02 +0200] Using SHA512 module Crypt::SHA [31/Jul/2024:22:26:02 +0200] PAM authentication enabled [31/Jul/2024:22:26:18 +0200] Reloading configuration`
sudo find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}'
Filename: /var/webmin/modules/authentic-theme/stats-server-555.log cat: /var/webmin/modules/authentic-theme/stats-server-555.log: Permission denied Filename: /var/webmin/modules/authentic-theme/stats-server-556.log cat: /var/webmin/modules/authentic-theme/stats-server-556.log: Permission denied
sudo find /var/webmin/modules/authentic-theme/ -name 'stats-server*' -print | xargs -I {} sh -c 'echo "Filename: {}"; cat {}'
Filename: /var/webmin/modules/authentic-theme/stats-server-555.log cat: /var/webmin/modules/authentic-theme/stats-server-555.log: Permission denied Filename: /var/webmin/modules/authentic-theme/stats-server-556.log cat: /var/webmin/modules/authentic-theme/stats-server-556.log: Permission denied
Try this instead:
sudo sh -c 'find /var/webmin/modules/authentic-theme/ -name "stats-server*" -print | xargs -I {} sh -c "echo Filename: {}; cat {}"'
websockets_/authentic-theme/ws-555=host=127.0.0.1 port=555 wspath=/ user=infras time=1722457578
Also, can you try logging in as root
rather than infras
user and see if it changes anything for you?
Filename: /var/webmin/modules/authentic-theme/stats-server-555.log
[31/Jul/2024:22:26:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 555
[31/Jul/2024:22:26:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40978 opened
[31/Jul/2024:22:26:28 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40978 closed
[31/Jul/2024:22:26:33 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40984 opened
[31/Jul/2024:22:26:35 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 40984 closed
[31/Jul/2024:22:26:39 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33354 opened
[31/Jul/2024:22:26:41 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33354 closed
[31/Jul/2024:22:26:43 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33366 opened
[31/Jul/2024:22:26:45 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33366 closed
[31/Jul/2024:22:26:49 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 opened
[31/Jul/2024:22:26:51 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 closed
[31/Jul/2024:22:27:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 59104 is closed due to inactivity
[31/Jul/2024:22:38:23 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 41370 opened
[31/Jul/2024:22:38:26 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 41370 closed
[31/Jul/2024:22:38:28 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39874 opened
[31/Jul/2024:22:38:30 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39874 closed
[31/Jul/2024:22:38:34 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39880 opened
[31/Jul/2024:22:38:36 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39880 closed
[31/Jul/2024:22:38:40 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 60156 opened
[31/Jul/2024:22:38:42 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 60156 closed
[31/Jul/2024:22:38:46 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 opened
[31/Jul/2024:22:38:48 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 closed
[31/Jul/2024:22:39:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43916 is closed due to inactivity
Filename: /var/webmin/modules/authentic-theme/stats-server-556.log
[31/Jul/2024:16:41:11 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket server is listening on port 556
[31/Jul/2024:16:41:15 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 55968 opened
[31/Jul/2024:16:41:17 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 55968 closed
[31/Jul/2024:16:41:17 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39326 opened
[31/Jul/2024:16:41:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39326 closed
[31/Jul/2024:16:41:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39332 opened
[31/Jul/2024:16:41:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39332 closed
[31/Jul/2024:16:41:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 opened
[31/Jul/2024:16:41:23 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 closed
[31/Jul/2024:16:41:36 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39346 is closed due to inactivity
[31/Jul/2024:17:17:59 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 opened
[31/Jul/2024:17:18:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 closed
[31/Jul/2024:17:18:14 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 36448 is closed due to inactivity
[31/Jul/2024:18:18:00 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 opened
[31/Jul/2024:18:18:02 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 closed
[31/Jul/2024:18:18:15 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 33068 is closed due to inactivity
[31/Jul/2024:19:18:01 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 opened
[31/Jul/2024:19:18:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 closed
[31/Jul/2024:19:18:16 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57132 is closed due to inactivity
[31/Jul/2024:20:18:03 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 opened
[31/Jul/2024:20:18:05 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 closed
[31/Jul/2024:20:18:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 39244 is closed due to inactivity
[31/Jul/2024:21:18:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 opened
[31/Jul/2024:21:18:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 closed
[31/Jul/2024:21:18:19 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 38576 is closed due to inactivity
[31/Jul/2024:22:18:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 opened
[31/Jul/2024:22:18:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 closed
[31/Jul/2024:22:18:21 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 43304 is closed due to inactivity
[31/Jul/2024:22:21:04 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57974 opened
[31/Jul/2024:22:21:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57974 closed
[31/Jul/2024:22:21:06 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57988 opened
[31/Jul/2024:22:21:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57988 closed
[31/Jul/2024:22:21:08 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57994 opened
[31/Jul/2024:22:21:10 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 57994 closed
[31/Jul/2024:22:21:10 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 58010 opened
[31/Jul/2024:22:21:12 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 58010 closed
[31/Jul/2024:22:21:16 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 opened
[31/Jul/2024:22:21:18 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 closed
[31/Jul/2024:22:21:31 +0200] [2001:41d0:305:2100::a0ca] /stats.cgi : WebSocket connection 56810 is closed due to inactivity