Closed TheFrankman closed 1 year ago
Good detective work @rjp-thijs - Shame you had to do it rather than NR giving us an update. This whole experience has been very damaging and confidence is completely lost with NR to keep their support up-to-date. The issue with OP Cache and Segfaults can't be used as an excuse as it should have been tested prior to release.
@zsistla - I see this got merged into dev 5 days ago... Any update on when the release will actually happen !?
Sorry bit I need this! I want subscribe New relic.
Worked for a few minutes, back to 'unknown'. PHP 8.1.5 Laravel 8.80.0 with opcache enabled.
up
Hi and thanks for your continued engagement.
We did identify a problem on the New Relic side and were able to roll that in to the most recent release 9.21.0.311
.
However, there are other issues occurring that we continue to investigate.
If opcache is required, the recommendation would be to use PHP 8.0 with Laravel 8+ since with Laravel 8+ and PHP 8.1 we observed frequent apache segfaults and frequent cases where the opcache simply stopped reporting information about classes.
At least some portion of the issue appears to be something to bring up with the laravel and/or PHP and/or apache teams as we observed the issues even with the PHP Agent disabled.
While there was a fix for an error on our side that was rolled into the latest release, there does seem to be some instability with Laravel and another process which is not related to NR as the following events were observed with NR disabled. If the underlying zend opcache is no longer to properly report on classes, the agent will be unable to function correctly. The remaining "unknown" issues were because the agent queried the opcache for a class and was told the class did not exist. Once in this state, for proper naming, the opcache needs to be reset.
Again, we've observed these issues with the NR agent disabled.
1) apache segfaults: The segfaults leave the opcache in a state that the php agent was unable to query about classes anymore and without the ability to query for classes in the opcache, we are unable to name the transaction properly (although the URI shows correctly and all children segments are properly labeled) and hence it shows as unknown. Once in this state, the opcache needs to be reset to provide the agent with the proper classes for naming again. This instability was also observed with the agent disabled. We tried with php 8.1.1, 8.1.2, 8.1.3, and 8.1.4 and while all exhibited segfaults, the frequency of segfaults decreased as the PHP version increased.
Sometimes only one apache process will segfault at a time, sometimes all of them will:
laravel_web | [Mon Apr 18 21:11:26.760412 2022] [core:notice] [pid 1] AH00052: child pid 33 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:29.764021 2022] [core:notice] [pid 1] AH00052: child pid 55 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:31.766644 2022] [core:notice] [pid 1] AH00052: child pid 56 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:34.772210 2022] [core:notice] [pid 1] AH00052: child pid 27 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:36.775568 2022] [core:notice] [pid 1] AH00052: child pid 28 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:38.778298 2022] [core:notice] [pid 1] AH00052: child pid 29 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:45.793312 2022] [core:notice] [pid 1] AH00052: child pid 63 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:47.798363 2022] [core:notice] [pid 1] AH00052: child pid 64 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:11:50.768259 2022] [core:notice] [pid 1] AH00052: child pid 65 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:12:06.805142 2022] [core:notice] [pid 1] AH00052: child pid 66 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:12:08.808704 2022] [core:notice] [pid 1] AH00052: child pid 67 exit signal Segmentation fault (11)
laravel_web | [Mon Apr 18 21:12:10.810495 2022] [core:notice] [pid 1] AH00052: child pid 68 exit signal Segmentation fault (11)
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.3 216704 31472 ? Ss 21:08 0:00 apache2 -DFOREGROUND
root 25 0.0 0.1 1157788 10676 ? Sl 21:08 0:00 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/
root 37 0.8 0.3 1382056 28448 ? Sl 21:08 0:01 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/
root 47 0.0 0.0 2420 528 pts/0 Ss 21:08 0:00 /bin/sh
www-data 64 3.0 0.3 222088 28928 ? R 21:11 0:00 apache2 -DFOREGROUND
www-data 65 0.0 0.0 216744 8356 ? S 21:11 0:00 apache2 -DFOREGROUND
www-data 66 0.0 0.0 216744 8356 ? S 21:11 0:00 apache2 -DFOREGROUND
www-data 67 0.0 0.0 216744 8356 ? S 21:11 0:00 apache2 -DFOREGROUND
www-data 68 0.0 0.0 216744 8356 ? S 21:11 0:00 apache2 -DFOREGROUND
www-data 69 0.0 0.0 216744 8356 ? S 21:11 0:00 apache2 -DFOREGROUND
root 70 0.0 0.0 6700 2988 pts/0 R+ 21:11 0:00 ps aux
2) There is another process (laravel? apache?) also dealing with the opcache in the same areas we are. This is apparent in the zend_get_resource_handle handle the agent gets. For all php version 8 and under, we get the same handle of 0 meaning no other entities requested a handle before us. For PHP 8.1, we get a higher numbered handle indicating the presence of another entity or profiler manipulating the opache in the same way the agent does. The agent and this entity appear to be clashing.
If another entity (including xdebug, APMs, etc.) is modifying the framework's default dispatching method, this causes incompatibilities and New Relic may not be able to detect or hook into the framework's dispatcher, and it will not be able to provide a meaningful transaction naming structure.
3) Laravel also exhibited instability under these conditions:
Fatal error: Uncaught Illuminate\Contracts\Container\BindingResolutionException: Target [Illuminate\Contracts\Http\Kernel] is not instantiable. in /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php:1089 Stack trace: #0 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(886): Illuminate\Container\Container->notInstantiable('Illuminate\\Cont...') #1 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(758): Illuminate\Container\Container->build('Illuminate\\Cont...') #2 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(851): Illuminate\Container\Container->resolve('Illuminate\\Cont...', Array, true) #3 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(694): Illuminate\Foundation\Application->resolve('Illuminate\\Cont...', Array) #4 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(836): Illuminate\Container\Container->make('Illuminate\\Cont...', Array) #5 /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php(1419): Illuminate\Foundation\Application->make('Illuminate\\Cont...') #6 /var/www/app/bootstrap/app.php(15): Illuminate\Container\Container->offsetGet('Illuminate\\Cont...') #7 /var/www/app/public/index.php(47): require_once('/var/www/app/bo...') #8 {main} thrown in /var/www/app/vendor/laravel/framework/src/Illuminate/Container/Container.php on line 1089
https://github.com/php/php-src/issues/8064
https://bugs.php.net/bug.php?id=81380
https://bugs.php.net/bug.php?id=81678
We will continue to investigate this issue.
its working when I restart nginx and fpm, but after some seconds....
@zsistla - I understand the segfault issue, and it's out of your control, but can you please let us know when we can expect the release for the issues on the newrelic side ?
Hi All,
I have only just stumbled upon this issue, however it appears that the latest release (9.21) of your PHP agent breaks PHP 7.4 reporting of transactions in Laravel 8, when it was working totally fine in version 9.19
We've only just discovered this today, and have had to go through and roll back all of our API webservers to previous version of the New Relic agent as every transaction was showing up as unknown.
same here with 9.21 in lumen 8 , and old version working well
I downgrade my PHP to 8.0 and now its working.
php: 7.4 lumen: 8 newrelic version: 9.21
Just to chime in here (as I've been chasing my tail over weird opcache bugs for the past 4 days) — it appears there's a known bug with PHP's opcache, which Laravel and Monolog seem to be triggering (though other frameworks and packages have also reported it too).
The bug is supposed to be fixed in PHP 8.1.6, which I believe is due to be released on May 12th. More info here: https://github.com/php/php-src/issues/8164
I mention this because of @zsistla's comment the other week:
If opcache is required, the recommendation would be to use PHP 8.0 with Laravel 8+ since with Laravel 8+ and PHP 8.1 we observed frequent apache segfaults and frequent cases where the opcache simply stopped reporting information about classes.
At least some portion of the issue appears to be something to bring up with the laravel and/or PHP and/or apache teams as we observed the issues even with the PHP Agent disabled.
I actually ended up installing New Relic for the first time to try and debug what was going on (so confirmed, it's happening with or without the agent disabled). My users were getting blank HTTP 500 pages, and Laravel was logging absolutely nothing because the opcache is refusing to invalidate a faulty compilation. Reloading PHP-FPM fixes it ~30% of the time (in my case, anyway), rebuilding the opcache ~50%, but it's totally random and very unpredictable.
Hopefully the PHP 8.1.6 release fixes this 🤞
@JackWH Thanks for the link! We'll try it out with 8.1.6 and see what happens 🤞🤞🤞🤞
@zsistla could you PLEASE give an indication as to when this commit will be merged to master !? https://github.com/newrelic/newrelic-php-agent/pull/419
Hi, #419 was merged into main and released with 9.21.0.311 April 26th which fixes a defect on our side. Unfortunately, as mentioned, the instability caused by an opcache defect with PHP 8.1 would still cause unknown
to occur because we depend on opcache for the details of cached functions and it will report the function does not exist even when it does in fact exist in the opcache. JackWH provided a link which documents some of the discussion around this: https://github.com/php/php-src/issues/8164. We have seen this cause segfaults with and without the PHP agent enabled.
Hopefully the PHP 8.1.6 release will solve the opcache defect.
So we've upgraded to v9.21.0.311 and PHP 8.1.6. Saldy we see no improvement and still almost all transactions are marked as Unknown.
@rjp-thijs same here, just this evening. On the bright side the 8.1.6 opcache fix has now started logging compilation errors... one of which, surprisingly, was this:
[2022-05-17 21:46:55] local.WARNING: Zend OPcache could not compile file vendor/newrelic/monolog-enricher/tests/integration/common/HandlerTest.php [{"Symfony\\Component\\Finder\\SplFileInfo":"vendor/newrelic/monolog-enricher/tests/integration/common/HandlerTest.php"}]
[2022-05-17 21:46:56] local.ERROR: Cannot redeclare NewRelic\Monolog\Enricher\newrelic_get_linking_metadata() (previously declared in vendor/newrelic/monolog-enricher/tests/unit/ProcessorTest.php:26) {"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Cannot redeclare NewRelic\\Monolog\\Enricher\
newrelic_get_linking_metadata() (previously declared in vendor/newrelic/monolog-enricher/tests/unit/ProcessorTest.php:26) at vendor/newrelic/monolog-enricher/tests/integration/common/HandlerTest.php:26)
(FYI @zsistla ^)
@zsistla — possibly (?) related to all this, but as I haven't seen it mentioned yet... Since PHP 8.1.6, I'm seeing this in /var/log/newrelic/php_agent.log
:
2022-05-17 21:26:11.737 +0000 (3144 3144) warning: User instrumentation from opcache: missing 'scripts' key in status information
If I had to guess, the New Relic agent is presumably trying to call opcache_get_status()
, and read from the scripts
key of the array? If so, then is this a bug in the agent — as I have opcache enabled for php-fpm
, but disabled for CLI requests (as seems to be standard practice, from what I understand).
...which might explain why New Relic is able to perfectly log all non-web transactions correctly, but it's only the web ones (i.e. through php-fpm
) which are all being tagged as "unknown".
🤔
Might be onto something here... After temporarily disabling the opcache with opcache.enable = 0
(though leaving the extension installed), New Relic is now correctly identifying web requests through php-fpm
.
Unfortunately /var/log/newrelic/php_agent.log
is filling up rapidly, with a new line logged for every single request:
2022-05-17 21:56:35.851 +0000 (13379 13379) warning: User instrumentation from opcache: opcache status information is not an array
So it seems like the agent is seeing the opcache extension is installed, trying to query for the status via CLI, but unable to find it (as it's switched off for CLI requests). Meanwhile, web requests are being served from the opcache, but the agent is unable to resolve them. It seems like it's assuming the opcache is "fully" enabled, just because the extension is installed — it's not checking whether opcache.enable = 1
, or more importantly opcache.enable_cli = 1
.
(To be honest I have almost no idea what I'm talking about, but maybe it'll help narrow things down 🙃).
Agent version: v9.21.0.311 PHP Version: 8.1.6
Freshly installed agent from newrelic UI instructions, same issue continues.
All transactions are unknown, no error reporting at all. I can see errors from laravel logs, but can't reflected to NewRelic UI.
Hi all,
For those who need transaction data ASAP, I just released a Laravel New Relic package which bypasses these unknown transaction issues 🚀
Hopefully this helps anyone here who needs visibility into their data until a proper solution is found. (I haven't mentioned anywhere in the README that the package only came about because of this issue, as it can be used standalone anyway, so I'll keep that between us!)
The package is plug-and-play, it should work right out of the box. Most importantly I think it should bypass most of the issues being described here (it did for me anyway 😄).
The package identifies transaction names and manages their lifecycles "manually", so New Relic doesn't need to interact with the opcache to detect which methods are being called. Transactions are automatically recorded for:
It filters out long-running processes and internal calls by default, but it's configurable so you can tailor it to your liking.
I've been using this for about a week in production now. Without the package, all my web transactions were unknown (but they worked for CLI, speculatively due to opcache differences). With the package installed, everything is easily-identifiable across the board.
Hope this helps 👍
/cc @iquad @rjp-thijs @TheFrankman @jeffcaulfield-morphsites
Thanks @JackWH , but problem is not transaction names, problem is not recording any error log, which is primary usage of an APM here :/
I'm experiencing the same issues reported above after installing OPcache. All Laravel transactions are showing up as /unknown
, and the errors make it to the laravel.log file but are not being reported in NewRelic. I have some non-Laravel PHP files on the same server that work just fine (transactions are properly named and errors are properly reported).
My NewRelic instrumentation is essentially useless in this state.
PHP (8.1.6-fpm) Zend OPcache (8.1.6) Upgrading the NewRelic PHP agent from 9.20.0.310 to 9.21.0.311 did not help.
@JackWH That package looks useful, I will try it out tomorrow.
To fix the error messages, I'm wondering if the newrelic_notice_error()
method still works. If so, we should be able to bind a new exception handler in your package that extends the base Exceptions/Handler adding that method to the report
function.
<?php
namespace JackWH\LaravelNewRelic;
use Illuminate\Foundation\Exceptions\Handler as BaseExceptionHandler;
class Handler extends BaseExceptionHandler
{
/**
* Report or log an exception.
*
* This is a great spot to send exceptions to Sentry, Bugsnag, etc.
*
* @param \Exception $exception
* @return void
*/
public function report(Throwable $exception)
{
newrelic_notice_error($exception);
parent::report($exception);
}
}
It's been 2.5 months and I guess it is not even prioritized since board is changing, new features are adding but that "huge" problem has no activity at all.
It was a great pleasure to work with NewRelic.
So long, and thanks for all the fish!
Same issue here with Laravel v9.16.0 and PHP v8.1.6
All transactions are "unknown"
@alitopaloglu we are using this middleware to workaround the issue
<?php
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
class NewRelicTransactionName
{
/**
* Handle an incoming request.
*
* @param Request $request
* @param Closure(\Illuminate\Http\Request): (\Illuminate\Http\Response|\Illuminate\Http\RedirectResponse) $next
* @return Response|RedirectResponse
*/
public function handle(Request $request, Closure $next)
{
$response = $next($request);
if (extension_loaded('newrelic')) { // Ensure PHP agent is available
if ($route = $request->route()) {
newrelic_name_transaction($route->getAction('controller'));
// or alternatively:
// newrelic_name_transaction($route->getName());
}
}
return $response;
}
}
@brettraminGTR Did that solution work?
Has anyone got errors reporting again?
@thomasdavis Yes, we were able to get errors reporting again with this function in the Handler:
public function report(Throwable $exception)
{
if ($this->shouldReport($exception) && App::environment('production')) {
newrelic_notice_error($exception);
}
parent::report($exception);
}
I haven't tested the following yet, but it is a bit cleaner and more in line with Laravels documentation for exception reporting:
public function register()
{
$this->reportable(function (Throwable $e) {
newrelic_notice_error($e);
});
}
@crishoj
@JackWH This issue:
Unfortunately
/var/log/newrelic/php_agent.log
is filling up rapidly, with a new line logged for every single request:2022-05-17 21:56:35.851 +0000 (13379 13379) warning: User instrumentation from opcache: opcache status information is not an array
Is addressed by https://github.com/newrelic/newrelic-php-agent/pull/447, which will be included in the next release.
See https://www.php.net/ChangeLog-8.php#8.1.7 Specifically, https://github.com/php/php-src/issues/8461
I'm going to try to upgrade to PHP 8.1.7 to see if the issue is resolved.
See https://www.php.net/ChangeLog-8.php#8.1.7 Specifically, php/php-src#8461
I'm going to try to upgrade to PHP 8.1.7 to see if the issue is resolved.
It probably won't. It didn't for me.
Until someone from New Relic says otherwise, I'm assuming the bug is still something related to my comment here. This suggests it can be fixed, by enabling opcache globally for both HTTP and CLI requests — can anyone confirm?
The agent doesn't appear to be checking whether the opcache is enabled for both CLI & HTTP requests, or just one at a time. The error message I mentioned suggests it's trying to resolve opcached code for a non-opcached request. It seems to be doing so simply because the opcache extension is installed, but it's not checking whether it's actually active for the instrumented request.
This is a common setup for Laravel applications, as Laravel Forge enables opcache for PHP-FPM only — which may explain why many of us are seeing this happen.
Hopefully someone from New Relic can address this for certain.
I'm seeing this happen on a Magento 2 site, not strictly Laravel. Opcache is enabled for both web and CLI. PHP 8.1.6 and 8.1.7 have the same issue.
On Wed, Jun 29, 2022, 2:08 PM Jack @.***> wrote:
Until someone from New Relic says otherwise, I'm assuming the bug is still something related to my comment here https://github.com/newrelic/newrelic-php-agent/issues/387#issuecomment-1129362072. This suggests it can be fixed, by enabling opcache globally for both HTTP and CLI requests — can anyone confirm?
The agent doesn't appear to be checking whether the opcache is enabled for both CLI & HTTP requests, or just one at a time. The error message I mentioned suggests it's trying to resolve opcached code for a non-opcached request. It seems to be doing so simply because the opcache extension is installed, but it's not checking whether it's actually active for the instrumented request.
This is a common setup for Laravel applications, as Laravel Forge enables opcache for PHP-FPM only — which may explain why many of us are seeing this happen.
Hopefully someone from New Relic can address this for certain.
— Reply to this email directly, view it on GitHub https://github.com/newrelic/newrelic-php-agent/issues/387#issuecomment-1170494976, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAER3RNFAWBWDDGTBWBP4WLVRS3OBANCNFSM5Q6PEGLA . You are receiving this because you commented.Message ID: @.***>
No news? We went an upgraded our newrelic php agent's from v9.20.0.310 that was working to 10.0.0.312 and everything is once again showing as unknown.
We are using Laravel 8.x and PHP 7.4 - but also see this on Laravel 9 and PHP 8.1
Please
This middleware works for me in Lumen
<?php
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\RedirectResponse;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
class NewRelicTransactionName
{
/**
* Handle an incoming request.
*
* @param Request $request
* @param Closure(\Illuminate\Http\Request): (\Illuminate\Http\Response|\Illuminate\Http\RedirectResponse) $next
* @return Response|RedirectResponse
*/
public function handle(Request $request, Closure $next)
{
$response = $next($request);
if (extension_loaded('newrelic')) { // Ensure PHP agent is available
newrelic_name_transaction($this->getTransactionName($request));
}
return $response;
}
/**
* Builds the transaction name. It will return the assigned controller action first, then the route name before
* falling back to just "index.php"
*
* @param Request $request
*
* @return string
*/
public function getTransactionName(Request $request)
{
$route = $request->route();
if (is_array($route)) {
// Try the assigned controller action
if (isset($route[1]) && isset($route[1]['uses'])) {
return $route[1]['uses'];
} // Try named routes
elseif (isset($route[1]) && isset($route[1]['as'])) {
return $route[1]['as'];
}
}
return 'index.php';
}
}
The lack of updates continue 😠
Hello everyone, as always we thank you for your invaluable feedback and inputs that help us build a better product. This is a known issue that our team is actively working to resolve and is part of our currently prioritized initiatives. We are hopeful to have a solution that resolves this as well as improves the PHP Agent soon. We will provide more information on completion as we hash out further details but would like to assure you that this is actively being worked on at present.
Any update on this issue?
PHP 7.4 is nearing end of life in 2 months, and a lot of folks are working on PHP 8.1 upgrades as a result. We're about to be forced into a version of PHP which will break NewRelic. Please update!
@theloveofcode I'm told the end of the quarter there will likely be a fix available (Q3/September)
@theloveofcode @trothwell-carsguide we are expecting a release sometime in October 2022 that will address this issue. We will update with more details as we near the release.
10 days ago, september; and now, October. @ak-war , sorry but your product is basically useless right now. Most of PHP users are using laravel/symfony with PHP8+ and as a APM, NewRelic is simply a brick on our servers. No error reporting, no tracing, nothing.
That issue is created on March, I mean, like 6 months from now, half a year... Core feature is not working at all, but what I see from boards, you are developing other parts.
It is like, produce a car which drivers can't drive it, but you are painting other colors and developing other mirror technologies.
Guys, this car is not going somewhere, there is issue on engine?!
If you need help, just tell us what's wrong and what you need, and we'll help you, otherwise, tell NewRelic to stop PHP part.
Hey guys wr need help! Here
Same here, but with PHP 8.0.22 and Shopware 6.4.14.0 (which uses Symfony 5.4)
Most of the transactions are unknown, especially the ones with Redis in it. Question being after skimming the Issue, is this related solely to 8.1 or 8+ ?
Do you realise your product is basically unusable right now?
@raulamoretti @quickbutik-ronflorax We're using this plugin https://github.com/newrelic/newrelic-php-agent/issues/387#issuecomment-1133747566 and it is working fine for us so far.
Same here, every transaction containing curl ends up as unknown since php 8.1 upgrade :|
I'm having the same problem, I have a site running Laravel 8 and another one with the same problem running Laravel 9.
Both using PHP 8.1.
Same situation here, any update would be much appreciated !
Update on this issue
The error rate being 100% was actually legitimate and I have resolved it BUT all transactions are still being labelled as unknown.
Description
1 day after switching to the new release all of the transactions are labeled as unknown and the error rate is 100% despite no errors being thrown.
I have opened a forum discussion for this but I have had no response : https://discuss.newrelic.com/t/php-new-relic-completely-broken-after-php-8-1-upgrade/179337
Relevant Logs / Console output
DAEMON STATUS
PHP AGENT LOGS
Daemon logs from today are empty
Your Environment
We are running out of the box newrelic.ini settings and config template
Additional context