Agent 10.22.0.12 & 11.2.0.15 on php 7.4 PHP 7.4.33 in agent/php_user_instrument.c

[das-peter]

Description

We've been seeing random stack failures on one of our last php 7.4 systems. Once core dumps were enabled we could see that in all of the core dumps the nr agent was involved. As we saw that the NR agent was involved we upgraded to the latest version of it 10.22.0.12 -> 11.2.0.15 but within hours we had another segfault. Below you'll find a excerpt of backtraces from the collected core dumps.

For the moment we decided to see if disabling the agent via newrelic.enabled stops the failures. If that doesn't work we'll completely disable the extension to see if that helps. Since the failures are random and it can take more than a week between occurrences this is a ongoing process.

Steps to Reproduce

We haven't found any specific steps which would trigger the issue. Sometimes it was working for a week without issues, then we had multiple failures within a couple of hours.

Relevant Logs / Console output

11.2.0.15

Agent - 11.2.0.15 -core.php-fpm.5978.e79a3bcdf9fd.1727959653.log

10.22.0.12

Agent - 10.22.0.12 -core.php-fpm.397963.7a8065e6c40b.1727937097.log Agent - 10.22.0.12 -core.php-fpm.399674.7a8065e6c40b.1727937092.log Agent - 10.22.0.12 - core.php-fpm.796057.14037af4a05c.1727859777.log

Your Environment

System-Info: Docker image based on official build: https://hub.docker.com/layers/library/php/7.4-fpm/images/sha256-7c6a2cba718f37c4b3ca3486476c7d3dc68dc3ea9b12bde9c57bd4b8d9479fc0?context=explore

PHP-Version

PHP 7.4.33 (cli) (built: Nov 15 2022 06:05:55) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies

PHP enabled Extensions

bcmath version 7.4.33
bz2 version 7.4.33
Core version 7.4.33
ctype version 7.4.33
curl version 7.4.33
date version 7.4.33
dom version 20031129
exif version 7.4.33
fileinfo version 7.4.33
filter version 7.4.33
ftp version 7.4.33
gd version 7.4.33
hash version 7.4.33
iconv version 7.4.33
imagick version 3.7.0
intl version 7.4.33
json version 7.4.33
libxml version 7.4.33
mbstring version 7.4.33
mysqli version 7.4.33
mysqlnd version mysqlnd 7.4.33
newrelic version 11.2.0.15
openssl version 7.4.33
pcntl version 7.4.33
pcre version 7.4.33
PDO version 7.4.33
pdo_mysql version 7.4.33
pdo_sqlite version 7.4.33
Phar version 7.4.33
posix version 7.4.33
rdkafka version 6.0.3
readline version 7.4.33
redis version 6.0.0
Reflection version 7.4.33
session version 7.4.33
SimpleXML version 7.4.33
soap version 7.4.33
sockets version 7.4.33
sodium version 7.4.33
SPL version 7.4.33
sqlite3 version 7.4.33
standard version 7.4.33
tidy version 7.4.33
tokenizer version 7.4.33
uploadprogress version 2.0.2
xml version 7.4.33
xmlreader version 7.4.33
xmlwriter version 7.4.33

Additional context

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-322302

[newrelic-php-agent-bot]

Thank you for your issue report. A member of the New Relic PHP Agent Team will review your report and endeavor to respond in a timely manner. Need troubleshooting help? Please review some of our common troubleshooting documentation, starting with generating debug logs. From there, additional resources and tips are available in our troubleshooting section, accessible from the left-hand panel of our documentation site. Additionally, please be sure to visit our Explorer's Hub, where other members of the community may have solved an issue similar to yours with the help of our Support team members. Have a question regarding support for an Operating System, PHP Version, Framework, or Library? Please take a look at our Compatibility Docs. Binary support for ARM-based CPU architecture is now available via the tarball installation method for PHP Versions 8.0+! General installation instructions can be found on our Installation Overview page. Need more information about an Agent INI setting? Our Configuration page contains an explanation for each INI setting and how to modify the defaults. Looking for something else? All New Relic PHP Agent Documentation can be found under our Introduction page.

[newrelic-php-agent-bot]

This issue has been identified as requiring additional information to debug. Due to the potentially sensitive nature of the information we may have to collect, such as PHPInfo and Debug Logs, please follow these steps to create a support case, where a member of our New Relic Support Team will work with you to gather the necessary information securely and help debug your issue. If your subscription level does not include technical support, the other option available to customers is to use New Relic’s community support channel: Explorers Hub. From there, a member of our New Relic Support Team will work with you to gather the necessary information securely and help debug your issue. For all available options on how to get support, please see these resources.

[workato-integration[bot]]

This issue requires more information and/or work.

[das-peter]

Added forum topic: https://forum.newrelic.com/s/hubtopic/aAXPh0000006W6rOAE/agent-1022012-112015-on-php-74-php-7433-in-agentphpuserinstrument

[das-peter]

Further evaluation was done directly together with Neil & Co from the NewRelic support team:

1. Enabled verbose debug of the agent:

   newrelic.loglevel = "verbosedebug"
   newrelic.daemon.loglevel = "debug"

2. Make certain that these two lines are not commented (you can adjust the path as appropriate):

   newrelic.logfile = "/var/log/newrelic/php_agent.log"
   newrelic.daemon.logfile = "/var/log/newrelic/newrelic-daemon.log"

3. Encountered following log entries: 3.1 Monolog 2024-10-14 06:40:00.313 +0000 (1346270 1346270) verbosedebug: Creating PHP Package 'monolog/monolog', version '1', source suggestion 2024-10-14 06:40:00.313 +0000 (1346270 1346270) verbosedebug: Suggesting package monolog/monolog 1 Indication that monolog v1 is installed. This is incompatible with the agent logforwarding, logforwarding was enabled by default "recently". There seems to be a bug that leads to segfaults. 3.1.1 Workaround: Disable Application Logging (newrelic.application_logging.enabled=false) Disable Logforwarding (newrelic.application_logging.forwarding.enabled=false) \ \ 3.2 Framework detection 2024-10-14 14:12:44.317 +0000 (3160 3160) verbosedebug: framework 'Symfony2' detected with /var/www/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/FrameworkBundle.php, which ends with symfony/bundle/frameworkbundle/frameworkbundle.php 2024-10-14 14:12:44.317 +0000 (3160 3160) debug: 'Symfony2' naming is 'unknown' Check if that detected framework matches with what you use - in our case it didn't match leading to trouble. 3.2.1 Workaround: Hardcode the framework (newrelic.framework = symfony4)

These issues most likely should not lead to a segfault in the agent - but at least there are workarounds one can implement once you know what you've to look for.

Shout out to Neil and the whole NewRelic support team for going through the ton of logs with us and providing helpful insights which allowed us to workaround the issues. It was a pleasure to do some bug-squashing with you.

Cheers, Peter