Segfault in php-fpm 8.1 on Linux

nono-gdv commented 2 years ago

Description

Hello,

Since migrating a big(ish) app on PHP 8.1, we get repeated segfaults, a few times per day (no more than 25 segfaults per day per server, for a few million hits per day per server). Here is a sample stack trace:

#0  0x00007f3ed8800556 in zend_accel_class_hash_copy (source=0x46736bb0, source=0x46736bb0, target=0x55ffcca9c330) at ./Zend/zend_types.h:1173
1173            return p->refcount;

(gdb) bt
#0  0x00007f3ed8800556 in zend_accel_class_hash_copy (source=0x46736bb0, source=0x46736bb0, target=0x55ffcca9c330) at ./Zend/zend_types.h:1173
#1  zend_accel_load_script (persistent_script=persistent_script@entry=0x46736a80, from_shared_memory=from_shared_memory@entry=1)
    at ./ext/opcache/zend_accelerator_util_funcs.c:230
#2  0x00007f3ed87eed26 in persistent_compile_file (type=<optimized out>, file_handle=<optimized out>) at ./ext/opcache/ZendAccelerator.c:2240
#3  persistent_compile_file (file_handle=<optimized out>, type=<optimized out>) at ./ext/opcache/ZendAccelerator.c:1951
#4  0x000055ffcc3edef8 in compile_filename (type=type@entry=2, filename=filename@entry=0x7f3ed210d820) at ./Zend/zend_language_scanner.c:707
#5  0x000055ffcc45ae40 in zend_include_or_eval (inc_filename_zv=<optimized out>, type=2) at ./Zend/zend_execute.c:4617
#6  0x000055ffcc4693ba in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at ./Zend/zend_vm_execute.h:38713
#7  0x000055ffcc48f357 in execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:59122
#8  0x000055ffcc41b46e in zend_call_function (fci=<optimized out>, fci_cache=<optimized out>) at ./Zend/zend_execute_API.c:908
#9  0x000055ffcc41b785 in zend_call_known_function (fn=0x7f3ed8c05cd8, object=<optimized out>, called_scope=<optimized out>, retval_ptr=retval_ptr@entry=0x0, 
    param_count=param_count@entry=1, params=params@entry=0x7fff83b4f4e0, named_params=0x0) at ./Zend/zend_execute_API.c:997
#10 0x000055ffcc31b66a in spl_perform_autoload (class_name=0x40e9f288, lc_name=0x40e9f2c8) at ./ext/spl/php_spl.c:433
#11 0x000055ffcc41a6b6 in zend_lookup_class_ex (name=name@entry=0x40e9f288, key=0x40e9f2c8, flags=flags@entry=512) at ./Zend/zend_execute_API.c:1141
#12 0x000055ffcc41ba70 in zend_fetch_class_by_name (class_name=0x40e9f288, key=<optimized out>, fetch_type=fetch_type@entry=512)
    at ./Zend/zend_execute_API.c:1601
#13 0x000055ffcc466dd7 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at ./Zend/zend_vm_execute.h:6778
#14 0x000055ffcc48fac0 in execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:56351
#15 0x000055ffcc41b46e in zend_call_function (fci=fci@entry=0x7fff83b4f770, fci_cache=<optimized out>, fci_cache@entry=0x7f3ed21b60d0)
    at ./Zend/zend_execute_API.c:908
#16 0x00007f3ed8754a0d in curl_write_header (data=data@entry=0x55ffccb729e0 "\r\n", size=size@entry=1, nmemb=nmemb@entry=2, ctx=0x7f3ed8cc2a80)
    at ./ext/curl/interface.c:1596
#17 0x00007f3ed86c572c in chop_write (conn=0x55ffcce9ea80, conn=0x55ffcce9ea80, olen=2, optr=0x55ffccb729e0 "\r\n", type=2) at sendf.c:623
#18 Curl_client_write (conn=0x55ffcce9ea80, type=2, ptr=0x55ffccb729e0 "\r\n", len=len@entry=2) at sendf.c:679
#19 0x00007f3ed86c3a72 in Curl_http_readwrite_headers (data=data@entry=0x55ffcce92f90, conn=conn@entry=0x55ffcce9ea80, nread=nread@entry=0x7fff83b4f960, 
    stop_reading=stop_reading@entry=0x7fff83b4f95f) at http.c:3405
#20 0x00007f3ed86d7778 in readwrite_data (comeback=0x7fff83b4f9eb, done=0x7fff83b4f9ea, didwhat=<synthetic pointer>, k=0x55ffcce93088, conn=0x55ffcce9ea80, 
    data=0x55ffcce92f90) at transfer.c:677
#21 Curl_readwrite (conn=0x55ffcce9ea80, data=data@entry=0x55ffcce92f90, done=done@entry=0x7fff83b4f9ea, comeback=comeback@entry=0x7fff83b4f9eb)
    at transfer.c:1294
#22 0x00007f3ed86e1933 in multi_runsingle (multi=multi@entry=0x55ffcce98a20, now=..., data=data@entry=0x55ffcce92f90) at multi.c:1933
#23 0x00007f3ed86e2bd9 in curl_multi_perform (multi=multi@entry=0x55ffcce98a20, running_handles=running_handles@entry=0x7fff83b4fb64) at multi.c:2218
#24 0x00007f3ed86d9742 in easy_transfer (multi=0x55ffcce98a20) at easy.c:686
#25 easy_perform (events=false, data=0x55ffcce92f90) at easy.c:780
#26 curl_easy_perform (data=0x55ffcce92f90) at easy.c:799
#27 0x00007f3ed8755b24 in zif_curl_exec (execute_data=0x7f3ed8c16750, return_value=0x7fff83b4fcc0) at ./ext/curl/interface.c:3063
#28 0x000055ffcc490eba in ZEND_DO_ICALL_SPEC_RETVAL_UNUSED_HANDLER () at ./Zend/zend_vm_execute.h:1235
#29 execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:55752
#30 0x000055ffcc49773c in zend_execute (op_array=0x7f3ed8c02000, return_value=0x0) at ./Zend/zend_vm_execute.h:60123
#31 0x000055ffcc429875 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at ./Zend/zend.c:1792
#32 0x000055ffcc3c601f in php_execute_script (primary_file=<optimized out>) at ./main/main.c:2538
#33 0x000055ffcc26f335 in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1914

I suspect the errors are opcache-related; here is our opcache configuration:

[opcache]
opcache.memory_consumption=512
opcache.max_accelerated_files=1000000
opcache.revalidate_freq=0
opcache.revalidate_path=1
opcache.huge_code_pages=0
opcache.file_update_protection=30
opcache.jit=disable

We are running Debian 10 and PHP 8.1.6, up-to-date from packages.sury.org.

PHP Version

PHP 8.1.6 (cli) (built: May 17 2022 16:49:19) (NTS)

Operating System

Debian 10

iluuu1994 commented 2 years ago

Hi @nono-gdv! Unfortunately it's very hard to understand this problem without more context. Assuming the code is closed source and can't be shared, you could try the following:

Run zbacktrace in gdb to get a PHP backtrace
Run frame 1 and then p (char *)persistent_script->script.filename->val to get the name of the compiled script

Maybe you can then share the given file if the problem can be isolated.

nono-gdv commented 2 years ago

Here you are.

The file is from Guzzle, this version: https://github.com/guzzle/guzzle/blob/7.4.3/src/Handler/HeaderProcessor.php

This is not consistent, however; segfaults sometimes happen in other files, even though Guzzle is a usual offender.

We are in the process of downgrading to PHP 8.0.19; it is a little early to tell for sure, but it looks like the segfaults are gone.

Reading symbols from /usr/sbin/php-fpm8.1...Reading symbols from /usr/lib/debug/.build-id/9a/0b98dd77386d65f38fe29707782edadbdc0057.debug...done.
done.
[New LWP 2329]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www                                                          '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f3ed8800556 in zend_accel_class_hash_copy (source=0x46736bb0, source=0x46736bb0, target=0x55ffcca9c330) at ./Zend/zend_types.h:1173
1173    ./Zend/zend_types.h: Aucun fichier ou dossier de ce type.
(gdb) zbacktrace
[0x7f3ed8c169f0] Composer\Autoload\includeFile("/space/www/kounga/data/releases/20220607140342/vendor/composer/../guzzlehttp/guzzle/src/Handler/HeaderProcessor.php") /space/www/kounga/data/releases/20220607140342/vendor/composer/ClassLoader.php:571 
[0x7f3ed8c16950] Composer\Autoload\ClassLoader->loadClass("GuzzleHttp\Handler\HeaderProcessor") /space/www/kounga/data/releases/20220607140342/vendor/composer/ClassLoader.php:428 
[0x7f3ed8c16880] GuzzleHttp\Handler\EasyHandle->createResponse() /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/EasyHandle.php:71 
[0x7f3ed8c167b0] GuzzleHttp\Handler\CurlFactory->GuzzleHttp\Handler\{closure}(object[0x7f3ed8c16800], "\15\12") /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:571 
[0x7f3ed8c16750] curl_exec(object[0x7f3ed8c167a0]) [internal function]
[0x7f3ed8c166b0] GuzzleHttp\Handler\CurlHandler->__invoke(object[0x7f3ed8c16700], array(10)[0x7f3ed8c16710]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php:44 
[0x7f3ed8c16600] GuzzleHttp\Handler\Proxy->GuzzleHttp\Handler\{closure}(object[0x7f3ed8c16650], array(10)[0x7f3ed8c16660]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php:28 
[0x7f3ed8c16550] GuzzleHttp\Handler\Proxy->GuzzleHttp\Handler\{closure}(object[0x7f3ed8c165a0], array(10)[0x7f3ed8c165b0]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php:48 
[0x7f3ed8c16470] GuzzleHttp\PrepareBodyMiddleware->__invoke(object[0x7f3ed8c164c0], array(10)[0x7f3ed8c164d0]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php:35 
[0x7f3ed8c163c0] GuzzleHttp\Middleware->GuzzleHttp\{closure}(object[0x7f3ed8c16410], array(10)[0x7f3ed8c16420]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Middleware.php:31 
[0x7f3ed8c16320] GuzzleHttp\RedirectMiddleware->__invoke(object[0x7f3ed8c16370], array(10)[0x7f3ed8c16380]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php:71 
[0x7f3ed8c16280] GuzzleHttp\Middleware->GuzzleHttp\{closure}(object[0x7f3ed8c162d0], array(10)[0x7f3ed8c162e0]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Middleware.php:63 
[0x7f3ed8c161f0] GuzzleHttp\HandlerStack->__invoke(object[0x7f3ed8c16240], array(10)[0x7f3ed8c16250]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/HandlerStack.php:75 
[0x7f3ed8c160a0] GuzzleHttp\Client->transfer(object[0x7f3ed8c160f0], reference) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Client.php:331 
[0x7f3ed8c15fc0] GuzzleHttp\Client->requestAsync("GET", object[0x7f3ed8c16020], array(11)[0x7f3ed8c16030]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Client.php:168 
[0x7f3ed8c15f30] GuzzleHttp\Client->request("GET", "listing?boutique_id=18&langue_id=1&gammesId=31084%2C25476&nbImage=2&from=0&categoriesId=15333&tmp=1654610655199", array(1)[0x7f3ed8c15fa0]) /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Client.php:187 
[0x7f3ed8c15eb0] GuzzleHttp\Client->get("listing?boutique_id=18&langue_id=1&gammesId=31084%2C25476&nbImage=2&from=0&categoriesId=15333&tmp=1654610655199") /space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/ClientTrait.php:44 
[0x7f3ed8c15dc0] App\Http\Controllers\Shopinvest\ProductController->listingDyn(object[0x7f3ed8c15e10]) /space/www/kounga/data/releases/20220607140342/app/Http/Controllers/Shopinvest/ProductController.php:1165 
[0x7f3ed8c15d40] Illuminate\Routing\Controller->callAction("listingDyn", array(1)[0x7f3ed8c15da0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Controller.php:54 
[0x7f3ed8c15ca0] Illuminate\Routing\ControllerDispatcher->dispatch(object[0x7f3ed8c15cf0], object[0x7f3ed8c15d00], "listingDyn") /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:45 
[0x7f3ed8c15c40] Illuminate\Routing\Route->runController() /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Route.php:261 
[0x7f3ed8c15bc0] Illuminate\Routing\Route->run() /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Route.php:204 
[0x7f3ed8c15ac0] Illuminate\Routing\Router->Illuminate\Routing\{closure}(object[0x7f3ed8c15b10]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Router.php:725 
[0x7f3ed8c15a30] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c15a80]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:141 
[0x7f3ed8c15980] Illuminate\View\Middleware\ShareErrorsFromSession->handle(object[0x7f3ed8c159d0], object[0x7f3ed8c159e0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php:49 
[0x7f3ed8c158a0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c158f0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c15800] Illuminate\Session\Middleware\StartSession->handleStatefulRequest(object[0x7f3ed8c15850], object[0x7f3ed8c15860], object[0x7f3ed8c15870]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php:121 
[0x7f3ed8c15740] Illuminate\Session\Middleware\StartSession->handle(object[0x7f3ed8c15790], object[0x7f3ed8c157a0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php:64 
[0x7f3ed8c15660] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c156b0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c155a0] App\Http\Middleware\Secure->handle(object[0x7f3ed8c155f0], object[0x7f3ed8c15600]) /space/www/kounga/data/releases/20220607140342/app/Http/Middleware/Secure.php:43 
[0x7f3ed8c154c0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c15510]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c15440] Illuminate\Pipeline\Pipeline->then(object[0x7f3ed8c15490]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:116 
[0x7f3ed8c15390] Illuminate\Routing\Router->runRouteWithinStack(object[0x7f3ed8c153e0], object[0x7f3ed8c153f0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Router.php:726 
[0x7f3ed8c15290] Illuminate\Routing\Router->runRoute(object[0x7f3ed8c152e0], object[0x7f3ed8c152f0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Router.php:703 
[0x7f3ed8c15220] Illuminate\Routing\Router->dispatchToRoute(object[0x7f3ed8c15270]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Router.php:667 
[0x7f3ed8c151b0] Illuminate\Routing\Router->dispatch(object[0x7f3ed8c15200]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Routing/Router.php:656 
[0x7f3ed8c15130] Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}(object[0x7f3ed8c15180]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:167 
[0x7f3ed8c150a0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c150f0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:141 
[0x7f3ed8c14fc0] App\Http\Middleware\SameOrigin->handle(object[0x7f3ed8c15010], object[0x7f3ed8c15020]) /space/www/kounga/data/releases/20220607140342/app/Http/Middleware/SameOrigin.php:11 
[0x7f3ed8c14ee0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14f30]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c14e30] App\Http\Middleware\Xss->handle(object[0x7f3ed8c14e80], object[0x7f3ed8c14e90]) /space/www/kounga/data/releases/20220607140342/app/Http/Middleware/Xss.php:37 
[0x7f3ed8c14d50] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14da0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c14c90] App\Http\Middleware\Lang->handle(object[0x7f3ed8c14ce0], object[0x7f3ed8c14cf0]) /space/www/kounga/data/releases/20220607140342/app/Http/Middleware/Lang.php:24 
[0x7f3ed8c14bb0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14c00]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c14ae0] Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle(object[0x7f3ed8c14b30], object[0x7f3ed8c14b40]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php:37 
[0x7f3ed8c14a00] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14a50]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c148b0] Illuminate\Cookie\Middleware\EncryptCookies->handle(object[0x7f3ed8c14900], object[0x7f3ed8c14910]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php:67 
[0x7f3ed8c147d0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14820]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c146f0] Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle(object[0x7f3ed8c14740], object[0x7f3ed8c14750]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php:86 
[0x7f3ed8c14610] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14660]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c14570] Illuminate\Http\Middleware\TrustProxies->handle(object[0x7f3ed8c145c0], object[0x7f3ed8c145d0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php:39 
[0x7f3ed8c14490] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c144e0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c143d0] App\Http\Middleware\ControllerBinding->handle(object[0x7f3ed8c14420], object[0x7f3ed8c14430]) /space/www/kounga/data/releases/20220607140342/app/Http/Middleware/ControllerBinding.php:23 
[0x7f3ed8c142f0] Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(object[0x7f3ed8c14340]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:180 
[0x7f3ed8c14270] Illuminate\Pipeline\Pipeline->then(object[0x7f3ed8c142c0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:116 
[0x7f3ed8c141f0] Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(object[0x7f3ed8c14240]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:142 
[0x7f3ed8c14150] Illuminate\Foundation\Http\Kernel->handle(object[0x7f3ed8c141a0]) /space/www/kounga/data/releases/20220607140342/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:111 
[0x7f3ed8c14020] (main) /space/www/kounga/data/releases/20220607140342/public/index.php:52 
(gdb) frame 1
#1  zend_accel_load_script (persistent_script=persistent_script@entry=0x46736a80, from_shared_memory=from_shared_memory@entry=1) at ./ext/opcache/zend_accelerator_util_funcs.c:230
230     ./ext/opcache/zend_accelerator_util_funcs.c: Aucun fichier ou dossier de ce type.
(gdb) p (char *)persistent_script->script.filename->val
$1 = 0x4108a648 "/space/www/kounga/data/releases/20220607140342/vendor/guzzlehttp/guzzle/src/Handler/HeaderProcessor.php"

nono-gdv commented 2 years ago

I can confirm the segfaults are gone. Since last logrotate (about 5am, so 12h ago, give or take), we had 317 segfault with PHP 8.1, 300 of them in opcache.so. We downgraded to PHP 8.0 two hours ago, not a single segfault since.

xPaw commented 2 years ago

I had the same crash yesterday:

#0  0x00007fea1f24b8af in zend_accel_class_hash_copy (source=0x416548f0, target=0x5560d11e5370) at ./Zend/zend_types.h:1173
1173    ./Zend/zend_types.h: No such file or directory.
(gdb) bt
#0  0x00007fea1f24b8af in zend_accel_class_hash_copy (source=0x416548f0, target=0x5560d11e5370) at ./Zend/zend_types.h:1173
#1  zend_accel_load_script (persistent_script=0x416547c0, from_shared_memory=1) at ./ext/opcache/zend_accelerator_util_funcs.c:230
#2  0x00005560d0cf95b9 in compile_filename (type=type@entry=8, filename=filename@entry=0x7fea1f656280) at ./Zend/zend_language_scanner.c:707
#3  0x00005560d0d6936a in zend_include_or_eval (inc_filename_zv=<optimized out>, type=8) at ./Zend/zend_execute.c:4617
#4  0x00005560d0d766aa in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at ./Zend/zend_vm_execute.h:38713
#5  0x00005560d0d9d396 in execute_ex (ex=0x41654a20) at ./Zend/zend_vm_execute.h:59122
#6  0x00005560d0d28e24 in zend_call_function (fci=<optimized out>, fci_cache=<optimized out>) at ./Zend/zend_execute_API.c:908
#7  0x00005560d0d29245 in zend_call_known_function (fn=0x7fea1f67c038, object=<optimized out>, called_scope=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=param_count@entry=1, params=params@entry=0x7fffc6eb4400, named_params=0x0) at ./Zend/zend_execute_API.c:997
#8  0x00005560d0c2e470 in spl_perform_autoload (class_name=0x40893108, lc_name=0x40893130) at ./ext/spl/php_spl.c:433
#9  0x00005560d0d280fc in zend_lookup_class_ex (name=name@entry=0x40893108, key=0x40893130, flags=flags@entry=512) at ./Zend/zend_execute_API.c:1141
#10 0x00005560d0d29592 in zend_fetch_class_by_name (class_name=0x40893108, key=<optimized out>, fetch_type=fetch_type@entry=512) at ./Zend/zend_execute_API.c:1601
#11 0x00005560d0d74187 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at ./Zend/zend_vm_execute.h:6778
#12 0x00005560d0d9c1ed in execute_ex (ex=0x41654a20) at ./Zend/zend_vm_execute.h:56351
#13 0x00005560d0da513d in zend_execute (op_array=0x7fea1f67b000, return_value=0x0) at ./Zend/zend_vm_execute.h:60123
#14 0x00005560d0d3761d in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at ./Zend/zend.c:1790
#15 0x00005560d0cd30c1 in php_execute_script (primary_file=primary_file@entry=0x7fffc6eb6b50) at ./main/main.c:2538
#16 0x00005560d0b8086d in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1914

It seems to be related to autoload, as OP's trace also has spl_perform_autoload.

The zbacktrace basically boils down to the require call after file_exists inside of spl_autoload_register callback. The string (path) looks okay to me in the trace.

brainexe commented 2 years ago

we also have the same random segfault with 8.1.7...also quite rare but overall having many failing processed per day. PHP8.0 downgrade is no option and we tried already different opcache config changes (optimizatzion level, disable jit) without success.

jirkace commented 2 years ago

@brainexe what about 8.1.8, did you try it? Did you try opcache.jit=1205?

brainexe commented 2 years ago

jit is disabled and php8.1.8 will be used next week on prod, then we will see 🤞

joelvh commented 2 years ago

@brainexe any luck with PHP 8.1.8?

brainexe commented 2 years ago

yes, still failing on production with php8.1.8, sadly no core dumps on this production maschines.

[01-Aug-2022 07:14:54] NOTICE: ready to handle connections
[01-Aug-2022 07:14:54] NOTICE: systemd monitor interval set to 10000ms
[01-Aug-2022 07:19:30] NOTICE: [pool ds] child 13109 exited with code 0 after 276.533705 seconds from start
[01-Aug-2022 07:19:30] NOTICE: [pool ds] child 14412 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14414 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 13107 exited with code 0 after 277.014571 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14415 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 13101 exited with code 0 after 277.040152 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14416 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 13106 exited with code 0 after 277.125439 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14417 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 13108 exited with code 0 after 277.139766 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14418 started
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 13100 exited with code 0 after 277.181764 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14419 started
[01-Aug-2022 07:19:31] WARNING: [pool ds] child 13103 exited on signal 11 (SIGSEGV) after 277.186114 seconds from start
[01-Aug-2022 07:19:31] NOTICE: [pool ds] child 14420 started
[01-Aug-2022 07:19:32] WARNING: [pool ds] child 13105 exited on signal 11 (SIGSEGV) after 277.646481 seconds from start

Aug  1 07:19:31 pt86w1 kernel: [3523335.430827] php-fpm8.1[13103]: segfault at 7f978b400040 ip 000055b834921187 sp 00007fffdf760bc8 error 4 in php-fpm8.1[55b834752000+2f3000]
Aug  1 07:19:31 pt86w1 kernel: [3523335.430833] Code: 1f 00 48 89 fe 48 89 d7 77 11 e9 a4 d2 ff ff 0f 1f 40 00 48 89 d7 e9 d8 d2 ff ff e9 c3 d3 ff ff 0f 1f 00 48 8b 05 51 ab 2a 00 <8b> 10 85 d2 0f 85 06 af e4 ff 48 89 fa 81 e2 ff ff 1f 00 74 54 48
Aug  1 07:19:32 pt86w1 kernel: [3523335.890661] php-fpm8.1[13105]: segfault at 7f978b400040 ip 000055b834921187 sp 00007fffdf760bc8 error 4 in php-fpm8.1[55b834752000+2f3000]
Aug  1 07:19:32 pt86w1 kernel: [3523335.890672] Code: 1f 00 48 89 fe 48 89 d7 77 11 e9 a4 d2 ff ff 0f 1f 40 00 48 89 d7 e9 d8 d2 ff ff e9 c3 d3 ff ff 0f 1f 00 48 8b 05 51 ab 2a 00 <8b> 10 85 d2 0f 85 06 af e4 ff 48 89 fa 81 e2 ff ff 1f 00 74 54 48

php -v
PHP 8.1.8 (cli) (built: Jul 11 2022 08:53:35) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.8, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.8, Copyright (c), by Zend Technologies

wow-apps commented 1 year ago

does anyone tried to set: opcache.huge_code_pages=0 ?

tsg1992 commented 1 year ago

On board with same issue. We didn't mind a few SIGSEGV 11 here and there until it rendered a whole server pretty much down. Every request to a specific app kept crashing PHP-FPM threads producing 502 Bad Gateway results. The only remediation was restarting PHP-FPM and it's now back to the usual but rare segfaults. JIT is enabled.

Platform: Ubuntu 22.04.2 LTS GNU/Linux 5.15.0-75-generic PHP/FPM: PHP 8.1.2-1ubuntu2.11, OPcache v8.1.2-1ubuntu2.11

opcache.blacklist_filename = no value opcache.consistency_checks = 0 opcache.dups_fix = Off opcache.enable = On opcache.enable_cli = Off opcache.enable_file_override = Off opcache.error_log = no value opcache.file_cache = no value opcache.file_cache_consistency_checks = On opcache.file_cache_only = Off opcache.file_update_protection = 2 opcache.force_restart_timeout = 180 opcache.huge_code_pages = Off opcache.interned_strings_buffer = 16 opcache.jit = 1235 opcache.jit_bisect_limit = 0 opcache.jit_blacklist_root_trace = 16 opcache.jit_blacklist_side_trace = 8 opcache.jit_buffer_size = 500000000 opcache.jit_debug = 0 opcache.jit_hot_func = 127 opcache.jit_hot_loop = 64 opcache.jit_hot_return = 8 opcache.jit_hot_side_exit = 8 opcache.jit_max_exit_counters = 8192 opcache.jit_max_loop_unrolls = 8 opcache.jit_max_polymorphic_calls = 2 opcache.jit_max_recursive_calls = 2 opcache.jit_max_recursive_returns = 2 opcache.jit_max_root_traces = 1024 opcache.jit_max_side_traces = 128 opcache.jit_prof_threshold = 0.005 opcache.lockfile_path = /tmp opcache.log_verbosity_level = 1 opcache.max_accelerated_files = 10000 opcache.max_file_size = 0 opcache.max_wasted_percentage = 10 opcache.memory_consumption = 256 opcache.opt_debug_level = 0 opcache.optimization_level = 0x7FFEBFFF opcache.preferred_memory_model = no value opcache.preload = no value opcache.preload_user = no value opcache.protect_memory = Off opcache.record_warnings = Off opcache.restrict_api = no value opcache.revalidate_freq = 0 opcache.revalidate_path = Off opcache.save_comments = On opcache.use_cwd = On opcache.validate_permission = Off opcache.validate_root = Off opcache.validate_timestamps = On

bukka commented 1 year ago

@tsg1992 Can you try to disable jit and see if it's still crashing?

maxsky commented 2 months ago

This problem from remirepo install version.

I tried if use this command install yum --enablerepo=remi-php81 install, it will be crash.

But now I install though compile, its running normal! I spend more than 20 hours to solve this problem. T_T...

I had a same problem, if disable opcache all Apps work normal, this is my config:

opcache.enable=1
opcache.enable_cli=0
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.use_cwd=1
opcache.validate_timestamps=1
opcache.revalidate_freq=10
opcache.save_comments=0
opcache.record_warnings=1
opcache.enable_file_override=0
opcache.dups_fix=1
opcache.huge_code_pages=0

I'm also tried disable jit or set jit to 1205, it still exited on signal 11 (SIGSEGV) and response 502.

Here is core file contents through gdb:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.1.al7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php-fpm...Reading symbols from /usr/lib/debug/usr/sbin/php-fpm.debug...done.
done.
[New LWP 12252]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `php-fpm: pool www              '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f565c87bb6c in zend_accel_class_hash_copy (source=0x4439deb0, target=0x5577f727d4d0) at /usr/src/debug/php-8.1.29/Zend/zend_types.h:1181
1181            return p->refcount;
(gdb) bt
#0  0x00007f565c87bb6c in zend_accel_class_hash_copy (source=0x4439deb0, target=0x5577f727d4d0) at /usr/src/debug/php-8.1.29/Zend/zend_types.h:1181
#1  zend_accel_load_script () at /usr/src/debug/php-8.1.29/ext/opcache/zend_accelerator_util_funcs.c:249
#2  0x00005577f52262a9 in compile_filename (type=type@entry=2, filename=filename@entry=0x7f565787ac80) at /usr/src/debug/php-8.1.29/Zend/zend_language_scanner.c:707
#3  0x00005577f5295e7a in zend_include_or_eval (inc_filename_zv=<optimized out>, type=2) at /usr/src/debug/php-8.1.29/Zend/zend_execute.c:4840
#4  0x00005577f52a309a in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /usr/src/debug/php-8.1.29/Zend/zend_vm_execute.h:38765
#5  0x00005577f52ca7b6 in execute_ex () at /usr/src/debug/php-8.1.29/Zend/zend_vm_execute.h:59187
#6  0x00005577f5255ef4 in zend_call_function () at /usr/src/debug/php-8.1.29/Zend/zend_execute_API.c:931
#7  0x00005577f52562e5 in zend_call_known_function () at /usr/src/debug/php-8.1.29/Zend/zend_execute_API.c:1020
#8  0x00005577f515aed0 in spl_perform_autoload (class_name=0x443493d8, lc_name=0x7f5657854280) at /usr/src/debug/php-8.1.29/ext/spl/php_spl.c:446
#9  0x00005577f52551cc in zend_lookup_class_ex (name=<optimized out>, key=key@entry=0x0, flags=flags@entry=0) at /usr/src/debug/php-8.1.29/Zend/zend_execute_API.c:1164
#10 0x00005577f52553e9 in zend_lookup_class (name=<optimized out>) at /usr/src/debug/php-8.1.29/Zend/zend_execute_API.c:1185
#11 0x00005577f5147c72 in reflection_class_object_ctor.constprop.0 () at /usr/src/debug/php-8.1.29/ext/reflection/php_reflection.c:3966
#12 0x00005577f52d1a08 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-8.1.29/Zend/zend_vm_execute.h:1761
#13 execute_ex () at /usr/src/debug/php-8.1.29/Zend/zend_vm_execute.h:55844
#14 0x00005577f52d24bd in zend_execute (op_array=0x7f565786c000, return_value=0x0) at /usr/src/debug/php-8.1.29/Zend/zend_vm_execute.h:60188
#15 0x00005577f5264630 in zend_execute_scripts () at /usr/src/debug/php-8.1.29/Zend/zend.c:1857
#16 0x00005577f51ffa91 in php_execute_script () at /usr/src/debug/php-8.1.29/main/main.c:2551
#17 0x00005577f5049ece in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-8.1.29/sapi/fpm/fpm/fpm_main.c:1935
(gdb) list
1176    # define GC_MAKE_PERSISTENT_LOCAL(p) \
1177            do { } while (0)
1178    #endif
1179
1180    static zend_always_inline uint32_t zend_gc_refcount(const zend_refcounted_h *p) {
1181            return p->refcount;
1182    }
1183
1184    static zend_always_inline uint32_t zend_gc_set_refcount(zend_refcounted_h *p, uint32_t rc) {
1185            p->refcount = rc;

php / php-src