Closed psytester closed 5 years ago
Is this related to the PR you just made? any chance you can PR against staging?
No, this issue is not related to the PR. I just saw it while manually counting my processes and memory consumption after changing to MPM worker.
Do you prefer to get the per process memory footprint based on ps
or based on pmap
?
Here I found a helpful explaination about ps
and pmap
output:
http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html
another usefull tool is memstat
Maybe both kinds of command output should be processed. At the end apache2buddy could show some more information with ps, pmap and /proc/meminfo details together to user that it's easier to get the hole picture or what should be analysed in detail by user
I beleive apache2budy doesnt properly support worker or event mode, its a hack at best - we only ever worked with prefork, If you can share some insights at the commandline I can take a look but I can't promise anything.
On Tue, 12 Feb 2019 at 19:44, psytester notifications@github.com wrote:
Here I found a helpful explaination about ps and pmap output:
http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html
another usefull tool is memstat
Maybe both kinds of command output should be processed. At the end apache2buddy could show some more information with ps, pmap and /proc/meminfo details together to user that it's easier to get the hole picture or what should be analysed in detail by user
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/richardforth/apache2buddy/issues/295#issuecomment-462906167, or mute the thread https://github.com/notifications/unsubscribe-auth/AFpUV-j_6GqwKwxJzVRU4mX7C5YJxB4rks5vMxm4gaJpZM4a2l_Y .
-- This message is considered confidential and is intended only for the recipient. If you receive this message in error please delete it immediately and disregard it's contents. Please also send an email to richard.forth@gmail.com to report the error.
It's not the best solution, but I added some additional checks to the script. Assuming a host should keep 20% of free RAM for OS internal tasks, all the Apache stuff (with additional mysql, java, redis, memcache, varnish, phpfpm and gluster) may consume up to 80% RAM only. Your current calculation is based on 100% RAM minus all those mentioned stuff. I canged to 80% RAM minus all those mentioned stuff.
If you like the new output, I can do a PR for it
Using the MPM prefork on a host where some other processes are consuming already almost all RAM:
[ -- ] httpd is currently using 13.43 MB of memory with 'ps -C ... -o rss' detection.
[ -- ] httpd is currently using 7.58 MB of memory with 'pmap -d ...' detection.
.......
--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: hostnameA ():
Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.
Your server's physical RAM: 1879 MB
80% of total usable RAM limit is: 1503 MB
Your server's physical current FREE RAM: 152 MB
Remaining Memory after other checked services considered: 1503 MB (for Apache itself and only up to 80% RAM!!!)
[ !! ] Your physical current FREE RAM is already lower than calculated max usable memory to Apache! 152 MB < 1503 MB
Apache's MaxRequestWorkers directive: 256 <--------- Current Setting
Apache MPM Model: prefork
Largest Apache process (by memory): 1 MB
[ !! ] Your MaxRequestWorkers setting is too low.
Your recommended MaxRequestWorkers setting is between 1056 and 1174. <------- Acceptable Range (10% of MAX)
Max potential memory usage: 327 MB
Percentage of TOTAL RAM allocated to Apache: 17.44 %
Percentage of REMAINING RAM allocated to Apache: 21.81 %
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.
Last 5 entries:
2019/02/13 13:08:43 Uptime: "0d 23h 24m 34s" Model: "Prefork" Memory: "1879 MB" Free Memory: "152 MB" MaxRequestWorkers: "256" Recommended: "1174" Smallest: "1.22 MB" Avg: "1.27 MB" Largest: "1.28 MB" Highest Pct Remaining RAM: "21.81%" (17.44% TOTAL RAM)
Using the MPM worker, the output looks like:
[ -- ] httpd is currently using 76.59 MB of memory with 'ps -C ... -o rss' detection.
[ -- ] httpd is currently using 2075.92 MB of memory with 'pmap -d ...' detection.
.......
--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: hostnameB ():
Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.
Your server's physical RAM: 1879 MB
80% of total usable RAM limit is: 1503 MB
Your server's physical current FREE RAM: 1249 MB
Remaining Memory after other checked services considered: 1503 MB (for Apache itself and only up to 80% RAM!!!)
[ !! ] Your physical current FREE RAM is already lower than calculated max usable memory to Apache! 1249 MB < 1503 MB
Apache's MaxRequestWorkers directive: 900 <--------- Current Setting
Apache MPM Model: worker
Largest Apache process (by memory): 519 MB
Apache appears to be running in worker mode.
Please check manually for backend processes such as PHP-FPM and pm.max_children.
Apache2buddy does not calculate maxclients for worker model.
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.
Last 5 entries:
2019/02/13 13:10:42 Uptime: "0d 23h 07m 00s" Model: "Worker" Memory: "1879 MB" Free Memory: "1249 MB" Maxclients: "900" Recommended: "N\A" Overall: "2075.99 MB" Smallest: "1.32 MB" Avg: "415.20 MB" Largest: "519.27 MB"
Those 80% lines, what are they? apache2buddy detects other services that consume RAM and take that away from the available RAM against which it can calculate maxclients for. Thats correct
I dont like the idea of adding an arbitrary 80% limit on it, but youre welcome to keep those changes in your own local custom copy if that helps you.
From my past with dealing with real high performance realtime based telco systems, the advice for proper system health was, that the system under load needs 20% space. Otherwise it might be possible that a new backgroud task can not be started or is delayed, or we are going into swapping, etc... That's why I added a simple 80% limit of total RAM.
You are detecting in apache2buddy only other services that you know or expect.
Here is my system in den wild and with all this research today and yesterday, I hit some bad issues on it. The system is running sine 120 days The nscd (name service caching daemon) is consuming ~ 786 MB The rsyslogd consumes ~ 328 MB
ps aux | egrep "[0-9]{6}|(USER)" | grep -v egrep
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 185032 4784 ? Ss 2018 13:46 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
nscd 931 0.0 0.1 804652 2588 ? Ssl 2018 7:33 /usr/sbin/nscd
root 933 0.0 0.1 335780 1980 ? Ssl 2018 1:00 /usr/sbin/rsyslogd -n
root 960 0.0 0.1 178516 3432 ? S 2018 152:43 /usr/bin/vmtoolsd
root 2289 0.0 1.9 129312 37996 ? Ssl 2018 24:46 /usr/bin/ruby.ruby2.1 /usr/bin/puppet agent --no-daemonize
root 5841 0.0 0.3 105008 7200 ? Ss Feb12 0:00 sshd: root@pts/0
root 32004 0.0 0.0 233576 384 ? S Feb11 0:00 (sd-pam)
root 32092 0.0 0.2 104796 4440 ? Ss Feb11 0:00 sshd: root@notty
2 GB RAM minus those big blocks of 2028 MB --> nothing left for real applications like Apache
If my system would be in a proper state, the calculation would be 80% of totoal RAM for the "real" applications and not the OS background.
Forget what I have written about the ps aux VSZ values and RAM consumption, I was on the wrong track.
First I restarted the nscd and it grows in VSZ value back to around those 80xxxx values.
I stopped the service at all and aoutput of free
was the same.
At the end I rebooted the system and got back my memory.
The root case was, that I'm running in ESX and my resource pool was reached. This and other VM hosts where ballooned and that's the reason for wrong free RAM calculation inside the host. I got more resources assigned and all ballooned hosts where relocated, but the wronf free RAM was visible. Only a reboot fixed that finally. More details found here: https://unix.stackexchange.com/questions/259659/high-memory-usage-but-no-process-is-using-it
Thanks for the update. As apache2buddy is predominantly used in web hosting we dont need to add a 20% arbitrary limit. If you want that for your own local copy thats fine. But Im not implementing that here in the upstream, it would be too disruptive..
I'm using MPM worker now and the usage of
ps -C httpd -o rss
will calculate a small amount which does not reflect the reality ?!?While getting the highest, lowest and average memory usage, I see a complete different footprint:
The RAM memory is really consumed now!
comand
top
is showingcommand
free
is showing the sameor the output of
meminfo
: