darenwelsh
commented
7 years ago /var/log/httpd/error_log has a bunch of Warning Interned string buffer overflow messages, but these begin long before the CPU usage issue that started today.
Open darenwelsh opened 7 years ago
darenwelsh
commented
7 years ago /var/log/httpd/error_log has a bunch of Warning Interned string buffer overflow messages, but these begin long before the CPU usage issue that started today.
darenwelsh
commented
7 years ago Per @freephile
try
<?php $wgMemoryLimit = 500000000; //Default is 50M. This is 500Min /opt/conf-meza/public/postLocalSettings.d/overrides.php to cure those interned string buffer overflow messages
I guess we can move that to a separate issue.
freephile
commented
7 years ago And increasing memory may not actually be the correct way to solve those Interned string messages. See https://superuser.com/questions/969386/php-5-6-what-does-opcache-interned-string-buffers-overflow-mean It looks like there are some php.ini settings which control the amount of memory in the APC opcache. Also, a link on that SU page shows a simple repo from Rasmus Ledorf for creating a status page for the APC cache. (it used to be included in the PEAR package, but now it's a separate file).
darenwelsh
commented
7 years ago Somewhere around 4:25 PM local time, the issue "resolved itself" - apache processes went back to normal usage.
freephile
commented
7 years ago wow, I was going to suggest using # lsof to perhaps see what might be the issue. Check the size of the Apache logs. If a log file grew to be extremely large due to some type of attack, maybe it was creating the CPU problem? Then if the log was rotated at 4:25, problem goes away.
darenwelsh
commented
7 years ago per @jamesmontalvo3 this may be related to #850 and we may need to manually uninstall the http-parser that may have been installed.
freephile
commented
7 years ago After further investigation into the Apache configuration of Meza (there is none, related to performance characteristics, so defaults are used), I filed Issue #867 which is very likely to be the source of the observed issue here. My suggestion would be to review https://discourse.equality-tech.com/t/how-do-i-optimize-apache/115 and use the given calculations, plus perl "Shortcut", to determine suitable configuration for your environment, then test with Apache Bench, or similar tool. The resolution to #867 would be to add Apache tuning to Meza.
Environment
Issue details
At 3:00 PM local time (Central Time Zone), one of our server administrators reported that our production VM was running at 100% CPU load (across all four CPUs). Per top, there were several apache processes running that were consuming the CPU load. So I ran
sudo apachectl restart. This seemed to solve the issue, but after 5-10 minutes, there were several apache processes running again and consuming the CPU load.The server admin noted that yum pushed package updates this morning. The following is from /var/log/yum.log: