Create timer error - segmentation fault

[deluxetom]

Hey there,

I've only had this issue on production and not locally or in a staging environment so I haven't been able to enable debug for this. My application gets around 40k requests per second, memory usage was fine but the CPU usage was spiking frequently making the instance crash. (ECS fargate with 2vCPU and 4G of RAM). With php-fpm I can have 3-4 instances to handle that much traffic and with FrankenPHP it needed more for some reason.

I have reverted to php-fpm for now but I just wanted to show you what I experienced.

As far as the app, it caches lists of images / videos to display. With frankenPHP I used array cache and with php-fpm I'm using APCu.

I'm not really sure how to debug segmentation faults in production without having to start a lot of ECS instances but I'm open to suggestions

Thanks!

Caddyfile

``` { frankenphp order php_server before file_server } :80, :{$PORT} { root * /app/{$PUBLIC_DIR} php_server header /assets/* { Access-Control-Allow-Origin "*" Cache-Control "public, max-age=31536000" } log { output stdout format console level error } push encode zstd gzip } ```

[dunglas]

This looks like a duplicate of #249.

Can you check if https://github.com/dunglas/frankenphp/issues/249#issuecomment-1752116059 fixes the issue? Thanks.

[deluxetom]

@dunglas I'll test that and let you know the results.

Are you aware of any issues with APCu? it seems the memory usage was a lot higher on my end, I wonder if the APCu cache is per process or shared across all of them.

[dunglas]

I don't know for APCu.

[SpencerMalone]

APCu is per unique parent / pre-fork PHP process execution. The author talked about it @ https://github.com/krakjoe/apcu/issues/121#issuecomment-136091336

If your process manager works like conventional FCGI/CGI then you will not be able to share (memory), if it works like FPM, and initializes PHP in a parent and forks child interpreters then you will have no problem.

If I am reading the interaction between go-land and c-land correctly, we're maintaining a thread pool and then calling php_execute_script in each thread pool, which I think means APCu won't be sharing memory across different worker threads.

[dunglas]

Do you know at which point APCu cache is initialized? We execute the PHP executor boot logic in the main PHP thread before creating children threads.

[SpencerMalone]

I do not, sadly, I did notice that and was unsure on exactly when you'd have to thread to make APCu share memory across different request workers. We could probably ask in APCu issues? I'm reading up in https://github.com/krakjoe/apcu/blob/master/TECHNOTES.txt and that makes it sound like in the MINIT phase, but there's some things in there that don't quite fit with my understanding of APCu, so maybe I've been living a lie for years, or maybe I'm not quite understanding the implication of those notes.

[SpencerMalone]

I just tested, APCu definitely works for franken in both worker and non-worker mode. Test repo @ https://github.com/SpencerMalone/apcu-test-prefork-franken. I'm not sure I understand why, but APCu definitely do be working for franken

[dunglas]

The segfault has been fixed by https://github.com/php/php-src/pull/12537

[CarlSinclair]

I'm having the same issue. After finally successfully building FrankenPHP with plugins, The only alteration I made to my Caddyfile was adding frankenphp and order php_server before file_server to my global options then commenting php_fastcgi and file_server in every site block and replacing them with php_server. I haven't configured it at all, sure, but this doesn't seem like normal/expected out of the box behavior.

Caddy starts normally, serves websites normally, but trying to run a page speed test makes it crash with the same error as this issue:

Update: it's crashing when serving websites as well. I didn't notice it earlier because I was testing a static PHP webpage which was likely cached by Cloudflare.

Jun 26 00:07:40 caddy[620113]: <br />
Jun 26 00:07:40 caddy[620113]: <b>Fatal error</b>:  Could not create timer: Resource temporarily unavailable (11) in <b>Unknown</b> on line <b>0</b><br />

Watching htop while running the test showed me spikes in CPU and memory usage caused by Caddy reaching over 95%. This has never happened on Apache or even Caddy with php_fastcgi even when running load tests with thousands of connects, let alone such a mundane single page load speed test.

Here's my version in case it's relevant: v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

(I find it extremely odd that it's not saying FrankenPHP in the version like some of my other builds, but it's correctly accepting the frankenphp global option so what do I know)

This is the command I used to build it:

CGO_ENABLED=1 XCADDY_GO_BUILD_FLAGS="-ldflags '-w -s'" xcaddy build     --output frankenphp     --with github.com/dunglas/frankenphp/caddy     --with github.com/dunglas/vulcain/caddy           --with github.com/caddy-dns/cloudflare             --with github.com/caddyserver/replace-response             --with github.com/greenpau/caddy-security             --with github.com/hslatman/caddy-crowdsec-bouncer             --with github.com/ggicci/caddy-jwt             --with github.com/ueffel/caddy-basic-auth-filter             --with github.com/pberkel/caddy-storage-redis             --with github.com/teodorescuserban/caddy-cookieflag             --with github.com/ltgcgo/floaty             --with github.com/baldinof/caddy-supervisor             --with github.com/quix-labs/caddy-image-processor 

And this is my systemd file:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=apache
Group=apache
#ExecStart=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
#ExecReload=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
ExecStart=/etc/caddy/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
ExecReload=/etc/caddy/caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
Environment="LD_LIBRARY_PATH=/etc/caddy/php/lib:$LD_LIBRARY_PATH" #custom built version of PHP as per documentation, version 8.3

[Install]
WantedBy=multi-user.target

Version of custom PHP build:

/etc/caddy/php/bin/php -v
PHP 8.3.0 (cli) (built: Jun 25 2024 00:59:46) (ZTS)
Copyright (c) The PHP Group
Zend Engine v4.3.0, Copyright (c) Zend Technologies

[CarlSinclair]

I'm not sure if this is relevant, but while troubleshooting I discovered that this custom build of PHP did not have an .ini file. I created one with the php.ini-production file from PHP's official Github repo and tried to point FrankenPHP at it (using PATH and environmental variables passed in the systemd service file) but it kept looking for it at an old directory (I built PHP in a directory then moved it to a different directory). So I created the directory it's looking for and placed the .ini file where it's looking for it and running php --ini shows that it's detecting it, but that hasn't fixed the problem.