search

richardforth / apache2buddy

apache2buddy
Apache License 2.0
385 stars 72 forks source link

Memory usage calculation 'ps -C httpd -o rss' vs. pmap -d #295

Closed psytester closed 5 years ago

psytester commented 5 years ago

I'm using MPM worker now and the usage of ps -C httpd -o rss will calculate a small amount which does not reflect the reality ?!?

[ -- ] httpd is currently using 81.22 MB of memory.

While getting the highest, lowest and average memory usage, I see a complete different footprint:

VERBOSE: Get 'high' memory usage
VERBOSE: Memory usage by PID 30112 is 1344K
VERBOSE: Memory usage by PID 30116 is 530296K
VERBOSE: Memory usage by PID 30117 is 530224K
VERBOSE: Memory usage by PID 30118 is 530200K
VERBOSE: Memory usage by PID 30353 is 530820K
VERBOSE: Get 'low' memory usage
VERBOSE: Memory usage by PID 30112 is 1344K
VERBOSE: Memory usage by PID 30116 is 530296K
VERBOSE: Memory usage by PID 30117 is 530224K
VERBOSE: Memory usage by PID 30118 is 530200K
VERBOSE: Memory usage by PID 30353 is 530820K
VERBOSE: Get 'average' memory usage
VERBOSE: Memory usage by PID 30112 is 1344K
VERBOSE: Memory usage by PID 30116 is 530296K
VERBOSE: Memory usage by PID 30117 is 530224K
VERBOSE: Memory usage by PID 30118 is 530200K
VERBOSE: Memory usage by PID 30353 is 530820K
[ -- ] The largest apache process is using 518.38 MB of memory
[ -- ] The smallest apache process is using 1.32 MB of memory
[ -- ] The average apache process is using 414.63 MB of memory

The RAM memory is really consumed now!

comand top is showing

KiB Mem:   1924272 total,  1769840 used,   154432 free,     2160 buffers

command free is showing the same

             total       used       free     shared    buffers     cached
Mem:       1924272    1774664     149608      11660       2168     376056
-/+ buffers/cache:    1396440     527832
Swap:      2103292     114652    1988640

or the output of meminfo:

cat /proc/meminfo
MemTotal:        1924272 kB
MemFree:          131572 kB
MemAvailable:     348940 kB
richardforth commented 5 years ago

Is this related to the PR you just made? any chance you can PR against staging?

psytester commented 5 years ago

No, this issue is not related to the PR. I just saw it while manually counting my processes and memory consumption after changing to MPM worker.

Do you prefer to get the per process memory footprint based on ps or based on pmap?

psytester commented 5 years ago

Here I found a helpful explaination about ps and pmap output: http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html

another usefull tool is memstat

Maybe both kinds of command output should be processed. At the end apache2buddy could show some more information with ps, pmap and /proc/meminfo details together to user that it's easier to get the hole picture or what should be analysed in detail by user

richardforth commented 5 years ago

I beleive apache2budy doesnt properly support worker or event mode, its a hack at best - we only ever worked with prefork, If you can share some insights at the commandline I can take a look but I can't promise anything.

On Tue, 12 Feb 2019 at 19:44, psytester notifications@github.com wrote:

Here I found a helpful explaination about ps and pmap output:

http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html

another usefull tool is memstat

Maybe both kinds of command output should be processed. At the end apache2buddy could show some more information with ps, pmap and /proc/meminfo details together to user that it's easier to get the hole picture or what should be analysed in detail by user

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/richardforth/apache2buddy/issues/295#issuecomment-462906167, or mute the thread https://github.com/notifications/unsubscribe-auth/AFpUV-j_6GqwKwxJzVRU4mX7C5YJxB4rks5vMxm4gaJpZM4a2l_Y .

-- This message is considered confidential and is intended only for the recipient. If you receive this message in error please delete it immediately and disregard it's contents. Please also send an email to richard.forth@gmail.com to report the error.

psytester commented 5 years ago

It's not the best solution, but I added some additional checks to the script. Assuming a host should keep 20% of free RAM for OS internal tasks, all the Apache stuff (with additional mysql, java, redis, memcache, varnish, phpfpm and gluster) may consume up to 80% RAM only. Your current calculation is based on 100% RAM minus all those mentioned stuff. I canged to 80% RAM minus all those mentioned stuff.

If you like the new output, I can do a PR for it

Using the MPM prefork on a host where some other processes are consuming already almost all RAM:

[ -- ] httpd is currently using 13.43 MB of memory with 'ps -C ... -o rss' detection.
[ -- ] httpd is currently using 7.58 MB of memory with 'pmap -d ...' detection.
.......
--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: hostnameA ():

Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

        Your server's physical RAM:                                   1879 MB
        80% of total usable RAM limit is:                             1503 MB
        Your server's physical current FREE RAM:                      152 MB
        Remaining Memory after other checked services considered:     1503 MB  (for Apache itself and only up to 80% RAM!!!)
[ !! ]  Your physical current FREE RAM is already lower than calculated max usable memory to Apache! 152 MB < 1503 MB
        Apache's MaxRequestWorkers directive:                         256      <--------- Current Setting
        Apache MPM Model:                                             prefork
        Largest Apache process (by memory):                           1 MB
[ !! ]  Your MaxRequestWorkers setting is too low.
        Your recommended MaxRequestWorkers setting is between 1056 and 1174.   <------- Acceptable Range (10% of MAX)
        Max potential memory usage:                                   327 MB
        Percentage of TOTAL RAM allocated to Apache:                  17.44  %
        Percentage of REMAINING RAM allocated to Apache:              21.81  %
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.

Last 5 entries:

2019/02/13 13:08:43 Uptime: "0d 23h 24m 34s" Model: "Prefork" Memory: "1879 MB" Free Memory: "152 MB" MaxRequestWorkers: "256" Recommended: "1174" Smallest: "1.22 MB" Avg: "1.27 MB" Largest: "1.28 MB" Highest Pct Remaining RAM: "21.81%" (17.44% TOTAL RAM)

Using the MPM worker, the output looks like:

[ -- ] httpd is currently using 76.59 MB of memory with 'ps -C ... -o rss' detection.
[ -- ] httpd is currently using 2075.92 MB of memory with 'pmap -d ...' detection.
.......
--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: hostnameB ():

Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

        Your server's physical RAM:                                   1879 MB
        80% of total usable RAM limit is:                             1503 MB
        Your server's physical current FREE RAM:                      1249 MB
        Remaining Memory after other checked services considered:     1503 MB  (for Apache itself and only up to 80% RAM!!!)
[ !! ]  Your physical current FREE RAM is already lower than calculated max usable memory to Apache! 1249 MB < 1503 MB
        Apache's MaxRequestWorkers directive:                         900      <--------- Current Setting
        Apache MPM Model:                                             worker
        Largest Apache process (by memory):                           519 MB
        Apache appears to be running in worker mode.
        Please check manually for backend processes such as PHP-FPM and pm.max_children.
        Apache2buddy does not calculate maxclients for worker model.
--------------------------------------------------------------------------------
A log file entry has been made in: /var/log/apache2buddy.log for future reference.

Last 5 entries:

2019/02/13 13:10:42 Uptime: "0d 23h 07m 00s" Model: "Worker" Memory: "1879 MB" Free Memory: "1249 MB" Maxclients: "900" Recommended: "N\A" Overall: "2075.99 MB" Smallest: "1.32 MB" Avg: "415.20 MB" Largest: "519.27 MB"
richardforth commented 5 years ago

Those 80% lines, what are they? apache2buddy detects other services that consume RAM and take that away from the available RAM against which it can calculate maxclients for. Thats correct

I dont like the idea of adding an arbitrary 80% limit on it, but youre welcome to keep those changes in your own local custom copy if that helps you.

psytester commented 5 years ago

From my past with dealing with real high performance realtime based telco systems, the advice for proper system health was, that the system under load needs 20% space. Otherwise it might be possible that a new backgroud task can not be started or is delayed, or we are going into swapping, etc... That's why I added a simple 80% limit of total RAM.

You are detecting in apache2buddy only other services that you know or expect.

Here is my system in den wild and with all this research today and yesterday, I hit some bad issues on it. The system is running sine 120 days The nscd (name service caching daemon) is consuming ~ 786 MB The rsyslogd consumes ~ 328 MB

ps aux | egrep "[0-9]{6}|(USER)" | grep -v egrep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.2 185032  4784 ?        Ss    2018  13:46 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
nscd       931  0.0  0.1 804652  2588 ?        Ssl   2018   7:33 /usr/sbin/nscd
root       933  0.0  0.1 335780  1980 ?        Ssl   2018   1:00 /usr/sbin/rsyslogd -n
root       960  0.0  0.1 178516  3432 ?        S     2018 152:43 /usr/bin/vmtoolsd
root      2289  0.0  1.9 129312 37996 ?        Ssl   2018  24:46 /usr/bin/ruby.ruby2.1 /usr/bin/puppet agent --no-daemonize
root      5841  0.0  0.3 105008  7200 ?        Ss   Feb12   0:00 sshd: root@pts/0
root     32004  0.0  0.0 233576   384 ?        S    Feb11   0:00 (sd-pam)
root     32092  0.0  0.2 104796  4440 ?        Ss   Feb11   0:00 sshd: root@notty

2 GB RAM minus those big blocks of 2028 MB --> nothing left for real applications like Apache

If my system would be in a proper state, the calculation would be 80% of totoal RAM for the "real" applications and not the OS background.

psytester commented 5 years ago

Forget what I have written about the ps aux VSZ values and RAM consumption, I was on the wrong track. First I restarted the nscd and it grows in VSZ value back to around those 80xxxx values. I stopped the service at all and aoutput of free was the same.

At the end I rebooted the system and got back my memory.

The root case was, that I'm running in ESX and my resource pool was reached. This and other VM hosts where ballooned and that's the reason for wrong free RAM calculation inside the host. I got more resources assigned and all ballooned hosts where relocated, but the wronf free RAM was visible. Only a reboot fixed that finally. More details found here: https://unix.stackexchange.com/questions/259659/high-memory-usage-but-no-process-is-using-it

richardforth commented 5 years ago

Thanks for the update. As apache2buddy is predominantly used in web hosting we dont need to add a 20% arbitrary limit. If you want that for your own local copy thats fine. But Im not implementing that here in the upstream, it would be too disruptive..