Open mathieudz opened 6 months ago
This may be related to https://github.com/dunglas/frankenphp/issues/578.
We need to find a way to understand what is blocking.
Another snapshot with a few observations:
cpu
0% and time
much lower than other thpool threads)Threads: 42 total, 0 running, 42 sleeping, 0 stopped, 0 zombie
%Cpu(s): 17.2 us, 1.8 sy, 0.0 ni, 80.7 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 64036.7 total, 2522.2 free, 27570.6 used, 43502.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 36466.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
112139 root 20 0 10.0g 6.5g 229132 S 99.9 10.4 6d+4h frankenphp
112524 root 20 0 10.0g 6.5g 229132 S 22.3 10.4 9,11 thpool-13
112522 root 20 0 10.0g 6.5g 229132 S 17.3 10.4 9,16 thpool-11
112517 root 20 0 10.0g 6.5g 229132 S 16.9 10.4 9,13 thpool-6
112514 root 20 0 10.0g 6.5g 229132 S 15.0 10.4 9,15 thpool-3
112526 root 20 0 10.0g 6.5g 229132 S 11.0 10.4 9,13 thpool-15
112520 root 20 0 10.0g 6.5g 229132 S 8.3 10.4 9,13 thpool-9
112513 root 20 0 10.0g 6.5g 229132 S 8.0 10.4 9,12 thpool-2
112516 root 20 0 10.0g 6.5g 229132 S 8.0 10.4 9,12 thpool-5
112511 root 20 0 10.0g 6.5g 229132 S 7.0 10.4 9,11 thpool-0
112515 root 20 0 10.0g 6.5g 229132 S 5.6 10.4 9,11 thpool-4
112525 root 20 0 10.0g 6.5g 229132 S 5.6 10.4 9,17 thpool-14
112527 root 20 0 10.0g 6.5g 229132 S 5.6 10.4 9,10 thpool-16
112512 root 20 0 10.0g 6.5g 229132 S 3.0 10.4 9,13 thpool-1
112523 root 20 0 10.0g 6.5g 229132 S 2.0 10.4 238:52.08 thpool-12
112519 root 20 0 10.0g 6.5g 229132 S 1.3 10.4 9,18 thpool-8
112487 root 20 0 10.0g 6.5g 229132 S 0.7 10.4 10:17.80 frankenphp
112482 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 33:49.28 frankenphp
112483 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 10:27.16 frankenphp
112488 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 10:30.22 frankenphp
112495 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 10:21.99 frankenphp
112529 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 6:21.43 frankenphp
112582 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 10:10.89 frankenphp
267509 root 20 0 10.0g 6.5g 229132 S 0.3 10.4 2:05.72 frankenphp
112484 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:44.28 frankenphp
112485 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:28.45 frankenphp
112486 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 0:00.00 frankenphp
112489 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:30.10 frankenphp
112490 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:13.67 frankenphp
112491 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:09.71 frankenphp
112492 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 0:00.00 frankenphp
112493 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:40.34 frankenphp
112494 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 0:00.00 frankenphp
112496 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 0:01.97 frankenphp
112518 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 145:54.01 thpool-7
112521 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 9,13 thpool-10
112528 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 9:44.18 frankenphp
112580 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:50.71 frankenphp
112581 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:27.89 frankenphp
112583 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 8:36.71 frankenphp
113843 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 10:21.58 frankenphp
1041453 root 20 0 10.0g 6.5g 229132 S 0.0 10.4 7:02.90 frankenphp
What happened?
Today, frankenphp used a limited number of workers, while 16 are defined. This caused severe slow down as these few workers were not enough to handle the load.
You can see here that most of the threads are idling:
Compare with threads after restart, where most of the thpool threads are active:
History of total CPU usage: you can clearly see where I restarted the docker container and frankenphp was able to use more resources:
It was in this state for about 1 hour.
I don't have any further diagnostics as I had to solve this problem urgently.
Build Type
Docker (Debian Bookworm)
Worker Mode
Yes
Operating System
GNU/Linux
CPU Architecture
x86_64
Relevant log output
No response