search

php / php-src

The PHP Interpreter
https://www.php.net
Other
38.31k stars 7.76k forks source link

SIGSEGV segfault crash with PHP 8.1.29 #15199

Closed archon810 closed 3 months ago

archon810 commented 4 months ago

Description

Hi,

We recently updated to OpenSUSE 15.6 (from 15.4) and are testing an update from PHP 8.0 to 8.1 (8.1.29 currently). On the test machine where we're testing, all of a sudden I'm seeing constant PHP8 crashes. After using gdb to see what's going on, they all seem to point to libpcre2-8-0, specifically pcre2_jit_match_8, php_pcre_replace_impl, php_pcre_replace, and php_replace_in_subject.

As has been suggested in other places like https://github.com/PCRE2Project/pcre2/issues/57, https://github.com/oerdnj/deb.sury.org/issues/1721, https://bugs.php.net/bug.php?id=81647, it seems like JIT is the culprit once again (the only other recent PHP crashes I've observed were related to opcache.jit being enabled).

Here's a sample backtrace from one such crash.

gdb php8 core.php.3000.pylon.1722366712
GNU gdb (GDB; devel:gcc) 14.2
...
Reading symbols from php8...
Reading symbols from /usr/lib/debug/usr/bin/php-8.1.29-lp156.2.1.x86_64.debug...
warning: Can't open file /var/run/nscd/dbUKm9bO (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libltdl.so.7.3.1 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libhogweed.so.6.9 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libnettle.so.8.9 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libcurl.so.4.8.0 (deleted) during file-backed mapping note processing
warning: Can't open file /usr/lib64/libpcre2-8.so.0.12.0 (deleted) during file-backed mapping note processing
[New LWP 3000]
warning: .dynamic section for "/usr/lib64/libpcre2-8.so.0" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libcurl.so.4" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libxcb.so.1" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libonig.so.4" is not at the expected address (wrong library or version mismatch?)
Missing separate debuginfo for /usr/lib64/libicuio.so.73.
The debuginfo package for this file is probably broken.
bMissing separate debuginfo for /usr/lib64/libicui18n.so.73.
The debuginfo package for this file is probably broken.
Missing separate debuginfo for /usr/lib64/libicuuc.so.73.
The debuginfo package for this file is probably broken.
Missing separate debuginfo for /usr/lib64/libicudata.so.73.
The debuginfo package for this file is probably broken.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `/usr/bin/php <SNIPPED BY archon810>'.
Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x000073d0084ecfc5 in pcre2_jit_match_8 (code=0x5bfa81d18570, subject=subject@entry=0x73cffd616d60 "SELECT 1", length=length@entry=8, start_offset=start_offset@entry=0, options=0, options@entry=1073741824, match_data=match_data@entry=0x5bfa81db53c0, mcontext=0x5bfa81db5350) at src/pcre2_jit_match.c:142
142       arguments.offset_limit = mcontext->offset_limit;
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.38-150600.14.5.1.x86_64 libcom_err2-debuginfo-1.47.0-150600.4.3.2.x86_64 libgcc_s1-debuginfo-13.3.0+git8781-150000.1.12.1.x86_64 libgcrypt20-debuginfo-1.10.3-150600.1.23.x86_64 libglib-2_0-0-debuginfo-2.78.6-150600.4.3.1.x86_64 libgpg-error0-debu
ginfo-1.47-150600.1.3.x86_64 libjitterentropy3-debuginfo-3.4.1-150000.1.12.1.x86_64 libltdl7-debuginfo-2.4.6-150000.3.8.1.x86_64 libmemcachedprotocol0-debuginfo-1.1.4-150600.2.2.x86_64 libmemcachedutil2-debuginfo-1.1.4-150600.2.2.x86_64 libonig4-debuginfo-6.7.0-150000.3.6.1.x86_64 libopenssl3-debuginfo-3.1.4-150
600.5.10.1.x86_64 libsodium23-debuginfo-1.0.18-150000.4.6.1.x86_64 libssh4-debuginfo-0.10.6-lp156.95.1.x86_64 libtiff5-debuginfo-4.0.9-150000.45.41.1.x86_64 libwebp7-debuginfo-1.0.3-150200.3.10.1.x86_64 libxcb1-debuginfo-1.13-150000.3.11.1.x86_64 libxml2-2-debuginfo-2.10.3-150500.5.17.1.x86_64 libxslt1-debuginfo
-1.1.39-lp156.146.1.x86_64 libzip5-debuginfo-1.10.1-150600.1.5.x86_64 php8-gd-debuginfo-8.1.29-lp156.2.1.x86_64 php8-intl-debuginfo-8.1.29-lp156.2.1.x86_64 php8-openssl-debuginfo-8.1.29-lp156.2.1.x86_64 php8-sockets-debuginfo-8.1.29-lp156.2.1.x86_64 php8-tokenizer-debuginfo-8.1.29-lp156.2.1.x86_64
(gdb) bt
#0  0x000073d0084ecfc5 in pcre2_jit_match_8 (code=0x5bfa81d18570, subject=subject@entry=0x73cffd616d60 "SELECT 1", length=length@entry=8, start_offset=start_offset@entry=0, options=0, options@entry=1073741824, match_data=match_data@entry=0x5bfa81db53c0, mcontext=0x5bfa81db5350) at src/pcre2_jit_match.c:142
#1  0x00005bfa816557b2 in php_pcre_replace_impl (pce=pce@entry=0x73cfff7b2040, subject_str=subject_str@entry=0x73cffd616d48, subject=subject@entry=0x73cffd616d60 "SELECT 1", subject_len=subject_len@entry=8, replace_str=replace_str@entry=0x73cfff97f360, limit=limit@entry=18446744073709551615,
    replace_count=0x7ffcac255208) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:1664
#2  0x00005bfa816560ce in php_pcre_replace (regex=<optimized out>, subject_str=0x73cffd616d48, subject=0x73cffd616d60 "SELECT 1", subject_len=8, replace_str=0x73cfff97f360, limit=18446744073709551615, replace_count=0x7ffcac255208) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:1606
#3  0x00005bfa81656acd in php_replace_in_subject (replace_count=0x7ffcac255208, limit=<optimized out>, subject=<optimized out>, replace_ht=0x0, replace_str=<optimized out>, regex_ht=<optimized out>, regex_str=<optimized out>) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2176
#4  preg_replace_common (is_filter=false, return_value=0x73cfe548d040, execute_data=0x73cfe548d110) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2314
#5  zif_preg_replace (execute_data=0x73cfe548d110, return_value=0x73cfe548d040) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/ext/pcre/php_pcre.c:2371
#6  0x000073d007de303d in xdebug_execute_internal (current_execute_data=0x73cfe548d110, return_value=0x73cfe548d040) at /tmp/pear/temp/xdebug/src/base/base.c:1024
#7  0x00005bfa816164ae in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1981
#8  0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#9  0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#10 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#11 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#12 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#13 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#14 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#15 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#16 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#17 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#18 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
#19 0x00005bfa81616419 in ZEND_DO_FCALL_SPEC_OBSERVER_HANDLER () at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:1946
#20 0x00005bfa8187760c in execute_ex (ex=0x5bfa82118800) at /usr/src/debug/php8-8.1.29-lp156.2.1.x86_64/Zend/zend_vm_execute.h:55852
...
this repeats for tens of thousands of lines - probably an endless loop

As you can see, it looks like the query was just SELECT 1 and whatever regex was used to parse it crashes the whole PHP.

I think in our code (using Wordpress), it happens here:

            //Specify a dummy query
            if ( $wpdb->db_connect( 'SELECT 1' ) ) {
                error_reporting( $error_reporting );

                $no_database = false;
                break;
            }

From what I can tell based on the messages in /var/log/messages, the pcre library loaded is libpcre2-8.so.0.13.0, which according to the package manager is version 10.44 (the latest available).

2024-08-01T11:02:55.361543-07:00 pylon kernel: [3516751.265471] php[19527]: segfault at 7fff87da0f98 ip 0000622dafa528ad sp 00007fff87da0fa0 error 6 in php[622daf600000+7cf000] likely on CPU 1 (core 1, socket 0)
2024-08-01T11:02:57.324535-07:00 pylon kernel: [3516753.226213] php[19707]: segfault at 7ffcf0b1cff8 ip 00007169714d1fe9 sp 00007ffcf0b1d000 error 6 in libpcre2-8.so.0.13.0[71697146e000+bf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:02:58.622204-07:00 pylon kernel: [3516754.517316] php[19534]: segfault at 7ffca2843fe8 ip 0000792ef127f422 sp 00007ffca2843fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:02:59.577390-07:00 pylon kernel: [3516755.481372] php[19614]: segfault at 7ffdc5062ff0 ip 00007b13d6ed8068 sp 00007ffdc5062fd0 error 6 likely on CPU 1 (core 1, socket 0)
2024-08-01T11:03:04.225560-07:00 pylon kernel: [3516760.127703] php[19524]: segfault at 7ffd6454ffd8 ip 00007d8ab5673f12 sp 00007ffd6454ffa0 error 6 likely on CPU 1 (core 1, socket 0)
2024-08-01T11:03:07.152392-07:00 pylon kernel: [3516763.054547] php[19706]: segfault at 7ffd871d6fe0 ip 000079beaa22c068 sp 00007ffd871d6fc0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:08.288678-07:00 pylon kernel: [3516764.192171] php[19705]: segfault at 7ffdc1141ff8 ip 0000799f3c373fe9 sp 00007ffdc1142000 error 6 in libpcre2-8.so.0.13.0[799f3c310000+bf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:03:09.114457-07:00 pylon kernel: [3516765.016904] php[19523]: segfault at 7fff5910afd8 ip 00007587ea441fdf sp 00007fff5910afa0 error 6 likely on CPU 4 (core 4, socket 0)
2024-08-01T11:03:14.294606-07:00 pylon kernel: [3516770.197875] php[19532]: segfault at 7ffeccc46fd8 ip 000063fb26c528ad sp 00007ffeccc46fe0 error 6 in php[63fb26800000+7cf000] likely on CPU 5 (core 5, socket 0)
2024-08-01T11:03:20.843902-07:00 pylon kernel: [3516776.745451] php[19525]: segfault at 7ffc3ae71fe8 ip 00007a2cb72c2fdf sp 00007ffc3ae71fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:24.383890-07:00 pylon kernel: [3516780.285203] php[19533]: segfault at 7ffdfb842fe8 ip 000079fb5177dfdf sp 00007ffdfb842fb0 error 6 likely on CPU 2 (core 2, socket 0)
2024-08-01T11:03:27.010622-07:00 pylon kernel: [3516782.913697] php[19704]: segfault at 7fff0dff3ff8 ip 00007d753d687fe9 sp 00007fff0dff4000 error 6 in libpcre2-8.so.0.13.0[7d753d624000+bf000] likely on CPU 4 (core 4, socket 0)
zypper info libpcre2-8-0
Loading repository data...
Reading installed packages...

Information for package libpcre2-8-0:
-------------------------------------
Repository     : openSUSE BuildService - devel:libraries:c_c++
Name           : libpcre2-8-0
Version        : 10.44-lp156.80.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:libraries:c_c++
Installed Size : 973.4 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : pcre2-10.44-lp156.80.1.src
Upstream URL   : https://www.pcre.org
Summary        : A library for Perl-compatible regular expressions
Description    : 
    The PCRE2 library is a set of functions that implement regular
    expression pattern matching using the same syntax and semantics
    as Perl 5.

    PCRE2 is a re-working of the original PCRE library to provide an entirely new
    API.

    This PCRE2 library variant supports 8-bit and UTF-8 strings.
    (See also libpcre2-16 and libpcre2-32)

I'm posting it here in case it's an issue with PHP itself.

I also posted to https://github.com/PCRE2Project/pcre2/issues/435. Tried disabling pcre.jit but it didn't work.

PHP Version

PHP 8.1.29

Operating System

OpenSUSE 15.6

devnexen commented 4 months ago 
warning: .dynamic section for "/usr/lib64/libpcre2-8.so.0" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libcurl.so.4" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib64/libxcb.so.1" is not at the expected address (wrong library or version mismatch?)
...

Quick question, with which opensuse release 8.1.29 comes from/had been built ?

The Leap 15.6 release incorporates several key software upgrades enhancing performance and security. It integrates Linux Kernel 6.4, which provides backports for some of latest hardware drivers, which offer performance enhancements. OpenSSL 3.1 becomes the new default and provides robust security features and updated cryptographic algorithms. Database management systems receive significant updates with MariaDB 10.11.6 and PostgreSQL 16. Redis 7.2 offers advanced data handling capabilities and the software stack is rounded out with PHP 8.2

archon810 commented 4 months ago

php8│ │8.1.29-lp156.2.1 │openSUSE BuildService - devel:languages:php

In this case, it came from this repo:

zypper addrepo -f  --name "openSUSE BuildService - devel:languages:php"               https://download.opensuse.org/repositories/devel:/languages:/php:/php81/openSUSE_Leap_15.6/ "download.opensuse.org-php"

We also tested PHP 8.2, but some Wordpress plugins we're using haven't been updated for it yet and output a lot of notices/warnings, so the plan was to move from 8.0 to 8.1 for now.

cmb69 commented 3 months ago

First, PHP 8.1 is in security mode now, so won't receive any regular bug fixes, but only security relevant fixes. This doesn't look like a security issue, though.

We also tested PHP 8.2, but some Wordpress plugins we're using haven't been updated for it yet and output a lot of notices/warnings, so the plan was to move from 8.0 to 8.1 for now.

That is unfortunate. Maybe you can urge the respective developers a bit to make their code compatible with PHP 8.2, and then switch to that version directly.

Tried disabling pcre.jit but it didn't work.

That would be an indication that PCRE's JIT is not the problem.

Anyhow, did you try running without Xdebug? Would that also crash?

github-actions[bot] commented 3 months ago

No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you.

archon810 commented 2 months ago

Anyhow, did you try running without Xdebug? Would that also crash?

Yes, it was also crashing without xdebug. All signs pointed to pcre.jit, so I'm stumped.

That is unfortunate. Maybe you can urge the respective developers a bit to make their code compatible with PHP 8.2, and then switch to that version directly.

We have been. Right now the biggest offender is https://github.com/google/site-kit-wp/issues/8331, though they have been closing some tickets about notices. I don't believe it's fully ready yet though.

cmb69 commented 2 months ago

From https://github.com/PCRE2Project/pcre2/issues/435:

Here's how it's set in both /etc/php8/apache/php.ini and /etc/php8/cli/php.ini:

pcre.jit=0

I verified that the value is set correctly:

php8 -r 'echo phpinfo();'|ack pcre.jit
pcre.jit => 0 => 0

That verification is irrelevant; you need to check when running a web request through Apache. Put a file containing

<?php
phpinfo();

somewhere in the docroot, and request that from a browser. I have the suspicion that neither disabling pcre.jit nor disabling Xdebug actually worked in your experiments so far.

archon810 commented 2 months ago

From the cores, I can actually see that all of them come from command line php scripts, not the web server (apache), which is why I was testing using the command line php.

cmb69 commented 2 months ago

Okay. Then please provide a stack backtrace without having loaded Xdebug and with pcre.jit=0.

archon810 commented 2 months ago

I think you may have been right, actually, and without xdebug the issue doesn't occur. What confused me earlier (apologies) is the crashes kept happening even after apache restarts, but after your comment I realized that those were all coming from php cli, and that those were managed by supervisor and continued to run in the background without restarting. Now that I restarted them, we haven't had a crash since (it's been 4 days).

Looks like the next steps are to re-enable pcre.jit and see if the crashes occur again. And regardless, I will try to report this to xdebug.

bradaric commented 2 months ago

I had the exact same issue - removing xdebug sorted it out even with pcre.jit enabled.

archon810 commented 2 months ago

@bradaric Ah, good to find someone else experiencing the same issue.

Do you have a reproducible case by chance? Xdebug bug reporting is pretty strict about that. https://xdebug.org/reporting-bugs

cmb69 commented 2 months ago

Maybe @derickr wants to have a look at this.

archon810 commented 2 months ago

@derickr Do you think you have enough to work with above in this case?

Edit: Ha, we both pinged him at the same time. I was just looking for the right Github user.

bradaric commented 2 months ago

Reading through those requirements - I'm not sure I can produce a script.

For me it only triggered on certain pages (using php8.3-fpm) and only with NewRelic PHP agent enabled (newrelic-php5). At first I though the agent was causing the trouble, but from their analysis ( https://docs.newrelic.com/docs/apm/agents/php-agent/troubleshooting/segmentation-faults/ ) it seems any observer would trigger it, if I understand that correctly...

I suppose I could try a clean PHP 8.3 system with just xdebug and the newrelic agent (or a test observer) and then do something memory-intensive in the script. I'll give it a try in the morning...

bradaric commented 2 months ago

Or is it xdebug actually clashing with newrelic? Would any 2 observers clash?

archon810 commented 2 months ago

FWIW we don't have newrelic enabled right now and those crashes still happen with xdebug.