[PHP7] Crash of php7.0-fpm with opcache and opcache.fast_shutdown

[mfn]

We're running 5.6/7.0 alongside and when enabling opcache with opcache.fast_shutdown=1 in PHP7 we experienced crashes almost instantly.

The log entry from /var/log/upstart/php7.0-fpm.log:

child 26210 exited on signal 11 (SIGSEGV) after 56.330280 seconds from start

Removing opcache.fast_shutdown=1 makes this problem go away. The same setting, on the same server, is activated for PHP 5.6 without problems.

xdebug is only installed but not enabled. I've only enabled the following php modules to trigger that crash (the minimum I needed to run the app and trigger the SIGSEGV):

• opcache
• pdo
• json
• pgsql
• pdo_pgsql

The crash doesn't happen for simple scripts. We experience this crash with CakePHP 2.8.3 (the latest in the 2.x line).

According to https://github.com/oerdnj/deb.sury.org/issues/300#issuecomment-196866807 I installed the *dbgsym packages and was able to produce this backtrace of a crashing fpm child:

Program received signal SIGSEGV, Segmentation fault.
0x00007f358e7434a0 in resetPQExpBuffer () from /usr/lib/x86_64-linux-gnu/libpq.so.5
(gdb) bt
#0  0x00007f358e7434a0 in resetPQExpBuffer () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#1  0x00007f358e736cf7 in ?? () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#2  0x00007f358e7386cb in PQsendQuery () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#3  0x00007f358e739d21 in PQexec () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#4  0x00007f358e95d733 in pgsql_stmt_dtor (stmt=0x7f358585f540) at /build/php7.0-1U3tsT/php7.0-7.0.5/ext/pdo_pgsql/pgsql_statement.c:64
#5  0x00007f358ed7cd6a in php_pdo_free_statement (stmt=0x7f358585f540) at /build/php7.0-1U3tsT/php7.0-7.0.5/ext/pdo/pdo_stmt.c:2316
#6  0x00007f3592162941 in zend_objects_store_del (object=0x7f358585f690) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_objects_API.c:182
#7  0x00007f3592128926 in _zval_dtor_func_for_ptr (p=<optimized out>) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_variables.c:109
#8  0x00007f3592128959 in i_zval_ptr_dtor (zval_ptr=0x7f358f6786f8) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_variables.h:58
#9  _zval_dtor_func_for_ptr (p=0x7f358f6786f0) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_variables.c:122
#10 0x00007f359215e014 in i_zval_ptr_dtor (zval_ptr=0x7f358f684228) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_variables.h:58
#11 zend_object_std_dtor (object=0x7f358f684100) at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_objects.c:69
#12 0x00007f35921625a0 in zend_objects_store_free_object_storage (objects=0x7f3585830008, objects@entry=0x7f359250b530 <executor_globals+816>)
    at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_objects_API.c:103
#13 0x00007f359211b913 in shutdown_executor () at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend_execute_API.c:357
#14 0x00007f359212a1c5 in zend_deactivate () at /build/php7.0-1U3tsT/php7.0-7.0.5/Zend/zend.c:967
#15 0x00007f35920ccee1 in php_request_shutdown (dummy=<optimized out>) at /build/php7.0-1U3tsT/php7.0-7.0.5/main/main.c:1826
#16 0x00007f3591fbf9c6 in main (argc=<optimized out>, argv=<optimized out>) at /build/php7.0-1U3tsT/php7.0-7.0.5/sapi/fpm/fpm/fpm_main.c:1996

Google lead led me to https://bugs.php.net/bug.php?id=71573 which is very recent, but explicitly says opcache is disabled already.

Very likely this is not related to your packages, but I'm not sure how to take it from here. I'm unable to come up with a small reproducible script :-/

So in spirit of the never-forgetting internet I created it here, even if it gets closed immediately maybe it helps someone else.

Packages from 7.0.5-2+deb.sury.org~trusty+1

ii  php-common                          1:36+deb.sury.org~trusty+1
ii  php-igbinary                        1.2.1-10-ge0e66b9+1.2.1-2+deb.sury.org~trusty+6
ii  php-redis                           2.2.7-389-g2887ad1+2.2.7-1+deb.sury.org~trusty+1
ii  php-xdebug                          2.4.0-1+deb.sury.org~trusty+1
ii  php5.6-cli                          5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-common                       5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-curl                         5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-fpm                          5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-gd                           5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-intl                         5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-json                         1.3.9-2+deb.sury.org~trusty+1
ii  php5.6-mbstring                     5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-mcrypt                       5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-mysql                        5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-opcache                      5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-pgsql                        5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-readline                     5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php5.6-xml                          5.6.20+dfsg-2+deb.sury.org~trusty+1
ii  php7.0-cli                          7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-common                       7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-curl                         7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-fpm                          7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-gd                           7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-intl                         7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-json                         7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-mbstring                     7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-mcrypt                       7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-mysql                        7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-opcache                      7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-pgsql                        7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-readline                     7.0.5-2+deb.sury.org~trusty+1
ii  php7.0-xml                          7.0.5-2+deb.sury.org~trusty+1

ps: thanks for your wonderful work!

[oerdnj]

@mfn Also try installing postgresql-X.Y-dbg (where X.Y is your postgresql version) to produce a full backtrace and I suggest you add the full backtrace to the upstream bug. That might help them debug the issue.

Also if you can reliably reproduce the bug, I suggest to add it to the bug as well even though it is not a simple script. A reproducible bug is always easier to fix.

[mfn]

Thanks for the encouragement. It's totally reproducible for me; here's an updated stacktrace with pgsql db:

Program received signal SIGSEGV, Segmentation fault.
resetPQExpBuffer (str=str@entry=0x7fb575b5fe10) at /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/pqexpbuffer.c:152
152     /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/pqexpbuffer.c: No such file or directory.
(gdb) bt
#0  resetPQExpBuffer (str=str@entry=0x7fb575b5fe10) at /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/pqexpbuffer.c:152
#1  0x00007fb568e99cf7 in PQsendQueryStart (conn=conn@entry=0x7fb575b5fab8) at /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/fe-exec.c:1352
#2  0x00007fb568e9b6cb in PQsendQuery (conn=conn@entry=0x7fb575b5fab8, query=query@entry=0x7fb575ba2550 "DEALLOCATE pdo_stmt_00000007")
    at /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/fe-exec.c:1115
#3  0x00007fb568e9cd21 in PQexec (conn=0x7fb575b5fab8, query=0x7fb575ba2550 "DEALLOCATE pdo_stmt_00000007")
    at /build/postgresql-9.5-DVsMQj/postgresql-9.5-9.5.2/build/../src/interfaces/libpq/fe-exec.c:1829
#4  0x00007fb5690c0733 in pgsql_stmt_dtor (stmt=0x7fb55dc72380) at /build/php7.0-kNWWO9/php7.0-7.0.5/ext/pdo_pgsql/pgsql_statement.c:64
#5  0x00007fb574f39d6a in php_pdo_free_statement (stmt=0x7fb55dc72380) at /build/php7.0-kNWWO9/php7.0-7.0.5/ext/pdo/pdo_stmt.c:2316
#6  0x00007fb57845b941 in zend_objects_store_del (object=0x7fb55dc724d0) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_objects_API.c:182
#7  0x00007fb578421926 in _zval_dtor_func_for_ptr (p=<optimized out>) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_variables.c:109
#8  0x00007fb578421959 in i_zval_ptr_dtor (zval_ptr=0x7fb575a98920) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_variables.h:58
#9  _zval_dtor_func_for_ptr (p=0x7fb575a98918) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_variables.c:122
#10 0x00007fb578457014 in i_zval_ptr_dtor (zval_ptr=0x7fb575a60928) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_variables.h:58
#11 zend_object_std_dtor (object=0x7fb575a60800) at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_objects.c:69
#12 0x00007fb57845b5a0 in zend_objects_store_free_object_storage (objects=0x7fb575b5fe10, objects@entry=0x7fb578804530 <executor_globals+816>)
    at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_objects_API.c:103
#13 0x00007fb578414913 in shutdown_executor () at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend_execute_API.c:357
#14 0x00007fb5784231c5 in zend_deactivate () at /build/php7.0-kNWWO9/php7.0-7.0.5/Zend/zend.c:967
#15 0x00007fb5783c5ee1 in php_request_shutdown (dummy=<optimized out>) at /build/php7.0-kNWWO9/php7.0-7.0.5/main/main.c:1826
#16 0x00007fb5782b89c6 in main (argc=<optimized out>, argv=<optimized out>) at /build/php7.0-kNWWO9/php7.0-7.0.5/sapi/fpm/fpm/fpm_main.c:1996

I'll add this information to the PHP report too.

[oerdnj]

I've pulled fix for PHP#71573 into ppa:ondrej/php-qa version php7.0_7.0.7~rc1-1+donate.sury.org~*+3, please try when it finishes building.

[mfn]

Thanks, I did => no change, immediately crashes as soon as opcache.fast_shutdown is enabled.

[mfn]

Haven't seen this crash anymore, also not using this PHP version anymore 🤷‍♀️ => closing