Investigate issues with php7.4-fpm on marconi.sown.org.uk

[drn05r]

php7.4-fpm seems to stop working properly. Athough systemctl reports no issues, the websites on marconi stop responding.

[cjsoftuk]

I have seen this before. It's usually disk/io related. The php-fpm pool processes end up sitting there with no available pool members to serve the request as everything is hung. I recommend turning on some timeouts if they aren't already (max_execution_time, and max_input_time are the two obvious starters for 5), and then investigating underlying hardware for IO issues.....

[drn05r]

@cjsoftuk I have upped the max_children from 5 to 16, which seemed to be the default on earlier versions of fpm. There are three modes:

• static
• dynamic
• ondemand It is currently set to dynamic. It looks like a number of the timeouts only work if you use the ondemand mode. Should I change to this mode before changing the max_execution_time and max_input_time settings?

[cjsoftuk]

I'm running dynamic in production on a number of hosts with no issues. max_exec_time and max_input_time are PHP settings, not FPM settings, so the FPM mode isn't really applicable here.

My experience suggests that these settings should reduce the impact, but may not eliminate it.

Looking at the timings, I think this is more IO-related than anything else as it generally happens overnight when backups may be running. Worth setting timeouts though in the short term as this may mitigate some impact.

[TimStallard]

hm, this has been going on for ages now, but I haven't got round to looking at it

last time I looked, it seemed like logrotate reloading things at midnight cleared whatever issue happened - so I wondered if something is causing apache/fpm workers to get stuck, then they eventually get killed at midnight

I think we had some behaviour like this with looping requests when we first switched from nginx to apache

[drn05r]

43 of the following error lines between 19:18 and 19:33 in /var/log/php7.4-fpm.log.1:

[12-May-2022 19:18:01] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

sown-monitor suggests the webiste first went down at 19:38 but this check only runs every 5 minutes.

I have updated /etc/php/7.4/fpm/php.ini and restart php7.4-fpm with the changes recommended by @cjsoftuk.

[drn05r]

Don't seem to have had any problems in the last month so I am tentatively going to say this is fixed. To confirm the following settings are now being used:

max_execution_time = 5 max_input_time = 5

The defaults were:

max_execution_time = 30 max_input_time = 60

[TimStallard]

this happened again, I've enabled apache forensic logging to try and see what causes it

[Tyler-Ward]

I think this can be closed as marconi has been replaced with a new server running docker