newrelic / newrelic-php-agent

The New Relic PHP Agent
https://opensource.newrelic.com/projects/newrelic/newrelic-php-agent
Apache License 2.0
119 stars 62 forks source link

All transactions unknown & 100% error rate post PHP8.1 upgrade (laravel v8) #387

Closed TheFrankman closed 1 year ago

TheFrankman commented 2 years ago

Update on this issue

The error rate being 100% was actually legitimate and I have resolved it BUT all transactions are still being labelled as unknown.

Description

1 day after switching to the new release all of the transactions are labeled as unknown and the error rate is 100% despite no errors being thrown.

I have opened a forum discussion for this but I have had no response : https://discuss.newrelic.com/t/php-new-relic-completely-broken-after-php-8-1-upgrade/179337

Relevant Logs / Console output

DAEMON STATUS

sudo service newrelic-daemon status
* newrelic-daemon.service - LSB: The New Relic Proxy Daemon
   Loaded: loaded (/etc/init.d/newrelic-daemon; generated)
   Active: active (exited) since Thu 2022-03-10 09:54:58 UTC; 1 weeks 0 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 17762 ExecStart=/etc/init.d/newrelic-daemon start (code=exited, status=0/SUCCESS)

Mar 10 09:54:58 [hidden] systemd[1]: Starting LSB: The New Relic Proxy Daemon...
Mar 10 09:54:58[hidden] newrelic-daemon[17762]: New Relic Daemon: newrelic-daemon already running
Mar 10 09:54:58 [hidden] systemd[1]: Started LSB: The New Relic Proxy Daemon.

PHP AGENT LOGS

2022-03-14 09:38:20.292 +0000 (18259 18259) warning: daemon connect(fd=16 uds=@newrelic) returned -1 errno=ECONNREFUSED. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: [Starting the PHP daemon (advanced) | New Relic Documentation](https://docs.newrelic.com/docs/apm/agents/php-agent/advanced-installation/starting-php-daemon-advanced/)
2022-03-14 09:39:01.420 +0000 (19119 19119) info: attempt daemon connection via ‘[@newrelic](https://discuss.newrelic.com/u/newrelic)’
2022-03-14 09:39:01.420 +0000 (19119 19119) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’[@newrelic](https://discuss.newrelic.com/u/newrelic)’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19119 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘hidden’]
2022-03-14 09:39:01.488 +0000 (19132 19132) info: attempt daemon connection via ‘[@newrelic](https://discuss.newrelic.com/u/newrelic)’
2022-03-14 09:39:01.488 +0000 (19132 19132) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’[@newrelic](https://discuss.newrelic.com/u/newrelic)’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19132 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘hidden’]
2022-03-14 09:39:01.550 +0000 (19145 19145) info: attempt daemon connection via ‘[@newrelic](https://discuss.newrelic.com/u/newrelic)’
2022-03-14 09:39:01.550 +0000 (19145 19145) info: New Relic 9.19.0.309 (“zomp” - “8536ad58490a”) [daemon=’[@newrelic](https://discuss.newrelic.com/u/newrelic)’ php=‘8.1.2’ zts=no sapi=‘cli’ pid=19145 ppid=19108 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os=‘Linux’ rel=‘4.19.0-17-amd64’ mach=‘x86_64’ ver=’#1 SMP Debian 4.19.194-2 (2021-06-21)’ node=‘hidden’]
2022-03-14 09:52:28.664 +0000 (19209 19209) error: TXNDATA failure: len=8180 errno=EPIPE
2022-03-14 09:52:30.232 +0000 (17106 17106) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:30.997 +0000 (19195 19195) error: TXNDATA failure: len=12332 errno=EPIPE
2022-03-14 09:52:31.026 +0000 (19193 19193) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:31.380 +0000 (19216 19216) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:32.692 +0000 (18355 18355) error: TXNDATA failure: len=11732 errno=EPIPE
2022-03-14 09:52:34.194 +0000 (19214 19214) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:37.828 +0000 (19217 19217) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:38.016 +0000 (19194 19194) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-14 09:52:39.934 +0000 (17863 17863) error: TXNDATA failure: len=12332 errno=EPIPE

Daemon logs from today are empty

Your Environment

Debian GNU/Linux 10 (buster)
PHP 8.1.2
Laravel Framework ^8.34

We are running out of the box newrelic.ini settings and config template

Additional context

image

image

zsistla commented 2 years ago

Hi, Thanks for bringing this to our attention.

There are a couple of things that caught our attention, and we'll need a little more information from you.

1) This line looks very odd: 2022-03-14 09:39:01.420 +0000 (19119 19119) info: attempt daemon connection via ‘[@newrelic](https://discuss.newrelic.com/u/newrelic)’ We'd expect to see something like this: 2022-03-16 23:45:52.080 +0000 (30413 30413) info: attempt daemon connection via '@newrelic'

This value is set: https://docs.newrelic.com/docs/apm/agents/php-agent/configuration/php-agent-configuration/#inivar-daemon-port Can you please let us know what the setting in your newrelic.ini file for newrelic.daemon.address

2) You also mention We are running out of the box newrelic.ini settings and config template Did you also configure the newrelic.cfg file? Your log snippet indicates the agent is expecting to automatically start the daemon, but the agent will not automatically start the daemon if it finds a newrelic.cfg file in the /etc/newrelic/ directory. Please verify if this does/does not exist.

3) please try `ps aux | grep newrelic" and see if the daemon is running. Try terminating all newrelic-daemon processes and restart your web service.

TheFrankman commented 2 years ago

Hello @zsistla

  1. My ini file does not have newrelic.daemon.address this has remained commented out.
  2. Yes it is configured and un-edited, just a copy of newrelic.cfg.template
  3. The result of ps-aux is
root     20042  0.0  0.0 758968 12660 ?        Sl   Mar14   0:07 /usr/bin/newrelic-daemon -c /etc/newrelic/newrelic.cfg --pidfile /var/run/newrelic-daemon.pid
root     20048  0.1  0.1 986504 28272 ?        Sl   Mar14   8:06 /usr/bin/newrelic-daemon -c /etc/newrelic/newrelic.cfg --pidfile /var/run/newrelic-daemon.pid -no-pidfile

I have not specifically killed the processes, but I have restarted the daemon and apache many times.

zsistla commented 2 years ago

Hi,

Thanks for the quick response.

From the ps command, we see /usr/bin/newrelic-daemon -c /etc/newrelic/newrelic.cfg --pidfile /var/run/newrelic-daemon.pid that shows the daemon was started with the newrelic.cfg file.

However, your log snippet indicates the agent is expecting to automatically start the daemon, but the agent will not automatically start the daemon if it finds a newrelic.cfg file in the /etc/newrelic/ directory.

Do you need to start the daemon manually for a specific reason? The standard installation for the PHP agent will automatically start the daemon if it detects that it is not running. The license key is configured for the agent in a PHP INI file, and it can be modified on a per-directory or per-virtual host basis. The daemon also is configured via the agent configuration (INI) file.

Because this newrelic.cfg file exists, the agent isn't automatically starting the daemon.

Can you please try to: 1) remove the newrelic.cfg file 2) terminate the daemon process 3) restart your web server

TheFrankman commented 2 years ago

Hello @zsistla,

Before I do this, since it's a production server. Can you please clarify why this would make a difference ? The New Relic Daemon is running, we can see this from my snippet where I checked the deamon status, and equally from ps aux. We also know that new relic is receiving a connection, there is throughput and transactions they are just confused in some way.

This seems to me to be an issue with new relics interpretation of the application rather than its run status.

Thanks, Frank.

zsistla commented 2 years ago

Hi Frank, Ah, got it regarding production server. The worry about the daemon is that it isn't producing logs. It's possible because of the way are you starting it that there is a permissions conflict with the logs directory. Let's try a few other things first. 1) delete newrelic.cfg 2) terminate newrelic-daemon 3) initiate some php traffic (can be done via web or CLI) 4) see if agent automatically restarts daemon

Also, is this a new installation of the PHP Agent? Was this something that was working prior to upgrading to the latest agent? If so, what was the previous agent version. We don't have official support for Laravel 8 yet, but if it was working with a prior agent version and isn't now, please let us know.

TheFrankman commented 2 years ago

It was working with Laravel 8 prior to the fresh installation of the new latest agent. I can't tell you what version it was on before i'm afraid. New Relic had been entirely broken for a couple of months while we were waiting for the extremely late release of PHP8.1 support.

I have followed the above steps. New result of ps aux is as follows :

sharpst+ 25019  0.0  0.1 537772 18604 ?        Sl   16:21   0:00 /usr/bin/newrelic-daemon --agent --logfile /var/log/newrelic/newrelic-daemon.log --port @newrelic --wait-for-port 0s --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
sharpst+ 25026  0.2  0.1 690520 22076 ?        Sl   16:21   0:00 /usr/bin/newrelic-daemon --agent --logfile /var/log/newrelic/newrelic-daemon.log --port @newrelic --wait-for-port 0s --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile

I am still seeing an average error rate of 100% and transactions are still all bundled into 'unknown' as name.

Latest logs from newrelic-deamon.log :

2022/03/17 16:21:01.166347 (25013) Info: New Relic daemon version 9.19.0.309-8536ad58490a [listen="@newrelic" startup=agent pid=25013 ppid=25012 uid=1001 euid=1001 gid=1001 egid=1001 runtime="go1.9.7" GOMAXPROCS=4 GOOS=linux GOARCH=amd64]
2022/03/17 16:21:01.236573 (25019) Info: New Relic daemon version 9.19.0.309-8536ad58490a [listen="@newrelic" startup=agent pid=25019 ppid=1 uid=1001 euid=1001 gid=1001 egid=1001 runtime="go1.9.7" GOMAXPROCS=4 GOOS=linux GOARCH=amd64]
2022/03/17 16:21:01.316546 (25026) Info: New Relic daemon version 9.19.0.309-8536ad58490a [listen="@newrelic" startup=agent pid=25026 ppid=25019 uid=1001 euid=1001 gid=1001 egid=1001 runtime="go1.9.7" GOMAXPROCS=4 GOOS=linux GOARCH=amd64]
2022/03/17 16:21:01.316661 (25026) Info: collector configuration is &{CAFile: CAPath: Proxy:}
2022/03/17 16:21:01.316887 (25026) Info: daemon listening on @newrelic
2022/03/17 16:21:04.268076 (25026) Info: Reporting to: https://rpm.eu.newrelic.com/accounts/2554422/applications/hidden-for-security
2022/03/17 16:21:04.268657 (25026) Info: app 'hidden for security' connected with run id 'hidden for security'

Latest lines from php_agent.log

2022-03-17 16:09:01.162 +0000 (24634 24634) info: attempt daemon connection via '@newrelic'
2022-03-17 16:09:01.162 +0000 (24634 24634) info: New Relic 9.19.0.309 ("zomp" - "8536ad58490a") [daemon='@newrelic'  php='8.1.2' zts=no sapi='cli'  pid=24634 ppid=24623 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os='Linux' rel='4.19.0-17-amd64' mach='x86_64' ver='#1 SMP Debian 4.19.194-2 (2021-06-21)' node='hidden-for-security']
2022-03-17 16:09:01.230 +0000 (24648 24648) info: attempt daemon connection via '@newrelic'
2022-03-17 16:09:01.230 +0000 (24648 24648) info: New Relic 9.19.0.309 ("zomp" - "8536ad58490a") [daemon='@newrelic'  php='8.1.2' zts=no sapi='cli'  pid=24648 ppid=24623 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os='Linux' rel='4.19.0-17-amd64' mach='x86_64' ver='#1 SMP Debian 4.19.194-2 (2021-06-21)' node='hidden-for-security']
2022-03-17 16:09:01.294 +0000 (24661 24661) info: attempt daemon connection via '@newrelic'
2022-03-17 16:09:01.294 +0000 (24661 24661) info: New Relic 9.19.0.309 ("zomp" - "8536ad58490a") [daemon='@newrelic'  php='8.1.2' zts=no sapi='cli'  pid=24661 ppid=24623 uid=0 euid=0 gid=0 egid=0 backtrace=yes startup=init os='Linux' rel='4.19.0-17-amd64' mach='x86_64' ver='#1 SMP Debian 4.19.194-2 (2021-06-21)' node='hidden-for-security']
2022-03-17 16:20:48.497 +0000 (24158 24158) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:49.698 +0000 (22637 22637) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:49.933 +0000 (22637 22637) warning: daemon connect(fd=16 uds=@newrelic) returned -1 errno=ECONNREFUSED. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: https://docs.newrelic.com/docs/apm/agents/php-agent/advanced-installation/starting-php-daemon-advanced/
2022-03-17 16:20:50.958 +0000 (24158 24158) warning: daemon connect(fd=16 uds=@newrelic) returned -1 errno=ECONNREFUSED. Failed to connect to the newrelic-daemon. Please make sure that there is a properly configured newrelic-daemon running. For additional assistance, please see: https://docs.newrelic.com/docs/apm/agents/php-agent/advanced-installation/starting-php-daemon-advanced/
2022-03-17 16:20:52.861 +0000 (22205 22205) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:55.409 +0000 (24232 24232) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:56.340 +0000 (24188 24188) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:56.487 +0000 (24779 24779) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:20:58.590 +0000 (24186 24186) error: TXNDATA failure: len=7932 errno=EPIPE
2022-03-17 16:20:59.987 +0000 (22222 22222) error: APPINFO failure: len=6340 errno=EPIPE
2022-03-17 16:21:01.404 +0000 (24045 24045) error: TXNDATA failure: len=11732 errno=EPIPE
2022-03-17 16:21:02.206 +0000 (24187 24187) error: APPINFO failure: len=6340 errno=EPIPE
mfulb commented 2 years ago

Just to be clear - things were working with a version of Laravel 8 (possibly different than is being used now) and a previous agent version and a previous PHP version? So at least two things have changed (agent version and PHP version)?

TheFrankman commented 2 years ago

To be abundantly clear

The Laravel version has not changed.

mfulb commented 2 years ago

The log you provided seems to indicate the PHP agent extension which loads into your application was unable to talk to the newrelic-daemon which handles communication with the backend. I think figuring out this problem is going to help us understand what is happening.

Having verbose log output might help - here are some instructions on setting this up. At this point I would recommend opening a case with support so we can track the case better.

TheFrankman commented 2 years ago

How do i open a case with support - I haven't found a way to do this.

mfulb commented 2 years ago

Sure I can help - our main portal for support is support.newrelic.com.

TheFrankman commented 2 years ago

@mfulb - I see no way of raising a support case. Only how to view existing cases. If you click through to i need more support it just redirects you to forums etc. Unfortunately all too typical.

TheFrankman commented 2 years ago

@mfulb - We have the newrelic.ini config stored in our ansible role, did the ini options change in the php8.1 release ?

TheFrankman commented 2 years ago

Disregard ^ I found the stub on this repo and it's identical (apart from actual values for license key, and a proper application name)

eduarguz commented 2 years ago

Any update on this?

We are facing the same problem, also there is a related discussion with no answer

ghost commented 2 years ago

We're really looking for a sollution on this as NewRelic now basically is unusable and it has been for a while. Can someone maybe try out my statement in https://github.com/newrelic/newrelic-php-agent/issues/392? That this appears to be Opcache related by disabling opcache?

I understand that that's not something you're able to do on a production environment, but maybe someone has this issue on a non-prod environment.

This knowledge won't fix this bug, but maybe help someone with knowledge fixing things :)

TheFrankman commented 2 years ago

@mfulb - It's time this was taken by the reins. New Relic is a paid service - We waited patiently for 3 months for PHP8.1 release and now people are still having issues. That means, some people have been paying for a broken product for 4 months at this point.

Have the team at least attempted to spin up NewRelic on PHP8.1 with Laravel installed ? Can you confirm that it works out of the box on a blank Laravel application ?

TheFrankman commented 2 years ago

I managed to get transaction names working for about 10 minutes, we had two versions of the new-relic.ini file (identical) which was clearly causing some issues, but even with that removed it went back to labelling all transactions as unknown.

I have some verbose error logging. I find the line debug: 'Laravel' naming is 'unknown' intriguing and obviously sending txnname='WebTransaction/Action/unknown' is relevant.

Even with this provided, i'd like confirmation that new relic have an instance of at least laravel 8.x running on php8.1 and can confirm that the agent works on a blank install.

2022-03-30 11:41:11.734 +0000 (20871 20871) verbosedebug: sending transaction message, len=12332
2022-03-30 11:41:11.734 +0000 (20871 20871) verbosedebug: post-deactivate processing done
2022-03-30 11:41:14.634 +0000 (21829 21829) verbosedebug: RINIT processing started
2022-03-30 11:41:14.634 +0000 (21829 21829) debug: 'WT_IS_FILENAME & SCRIPT_FILENAME' naming is '/index.php'
2022-03-30 11:41:14.634 +0000 (21829 21829) verbosedebug: RINIT processing done
2022-03-30 11:41:14.635 +0000 (21829 21829) debug: detected framework = 'Laravel'
2022-03-30 11:41:14.635 +0000 (21829 21829) debug: 'Laravel' naming is 'unknown'
........
........
........
2022-03-30 11:41:14.773 +0000 (21829 21829) verbosedebug: Skipping instrumented function: pid mismatch, got 20693, expected 21829
2022-03-30 11:41:14.776 +0000 (21829 21829) verbosedebug: Skipping instrumented function: pid mismatch, got 20693, expected 21829
2022-03-30 11:41:14.776 +0000 (21829 21829) verbosedebug: Skipping instrumented function: pid mismatch, got 20693, expected 21829
2022-03-30 11:41:14.777 +0000 (21829 21829) verbosedebug: RSHUTDOWN processing started
2022-03-30 11:41:14.777 +0000 (21829 21829) verbosedebug: request_uri='/api/v1/feeds/d33a2a72-73c8-4c30-8a07-e0e5311d9381'
2022-03-30 11:41:14.777 +0000 (21829 21829) verbosedebug: RSHUTDOWN processing done
2022-03-30 11:41:14.777 +0000 (21829 21829) verbosedebug: post-deactivate processing started
2022-03-30 11:41:14.777 +0000 (21829 21829) verbosedebug: /proc/self read: 16809 of 4096-byte pages
2022-03-30 11:41:14.777 +0000 (21829 21829) debug: txn naming freeze
2022-03-30 11:41:14.778 +0000 (21829 21829) verbosedebug: sending txnname='WebTransaction/Action/unknown' agent_run_id=hidden segment_count=308 duration=143351 threshold=2000000 priority=1.923335
2022-03-30 11:41:14.778 +0000 (21829 21829) verbosedebug: sending transaction message, len=12332
2022-03-30 11:41:14.778 +0000 (21829 21829) verbosedebug: post-deactivate processing done

Here are the processes for good measure :

hidden 22348  0.0  0.0 609936 13088 ?        Sl   11:37   0:00 /usr/bin/newrelic-daemon --agent --logfile /var/log/newrelic/newrelic-daemon.log --loglevel debug --port @newrelic --wait-for-port 0s --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true
hidden 22354  0.1  0.1 837984 23364 ?        Sl   11:37   0:02 /usr/bin/newrelic-daemon --agent --logfile /var/log/newrelic/newrelic-daemon.log --loglevel debug --port @newrelic --wait-for-port 0s --define utilization.detect_aws=true --define utilization.detect_azure=true --define utilization.detect_gcp=true --define utilization.detect_pcf=true --define utilization.detect_docker=true -no-pidfile
ghost commented 2 years ago

I'd like to report that our logging looks identical to the logging above. What I did notice were the "Skipping instrumented function" lines. Whenver we have a request that IS working, these line's dont appear.

TheFrankman commented 2 years ago

@rjp-thijs - Are you by any chance using redis-sentinel ?

TheFrankman commented 2 years ago

Ok, so confession time, the error rate being 100% was actually the fault of an extension we had for Redis Sentinel that was firing a deprecation on every request, so although the site was fully functional New Relic was registering the error rate as 100%. This was by bad because these errors were not in php errors, or laravels errors so i disregarded them as bogus.

The transactions labelled as 'unknown' however continues to be an issue. Every time i restart apache (and therefore new-relic) it works for a few minutes then transactions go back to being grouped under unknown.

ghost commented 2 years ago

We're not using redis-sentinel. What you've described above matches our problem. I can't say it has to do with reboots. For us it almost looks "random", but around 90% of the requests ends up "unknown". But as I wrote in the other thread, it happens that the a transaction might end up in unknown, and if the same request happens again it might(!) end up as a named tansaction. More often it will still go to unknown.

As you can see on the screenshot below the route /admin/shipments (aka admin.shipments.index) goes most often to "unknown" but has results as a named entry. Screenshot from 2022-03-30 19-40-54

ghost commented 2 years ago

Also one very intresting thing to add (which again makes me more think this is happening due to Opcache). We have two services, the one thats used in nginx and one that monitors all our console commands / terminal. The one that monitores nginx has the problem with naming almost all transactions as unknown and has opcache enabled. The one that monitors console commands doesn't name anything unknown.

crishoj commented 2 years ago

The one that monitores nginx has the problem with naming almost all transactions as unknown and has opcache enabled. The one that monitors console commands doesn't name anything unknown.

Exactly what I'm observing as well.

Frankly, New Relic is testing my patience as a paying customer. Might be time to consider other options.

raulamoretti commented 2 years ago

Hey guys, here we have the same problem.

All transactions are unknown.

Laravel 8 with php 8.1. we use naming in routes api.php

image
TheFrankman commented 2 years ago

Anyone know how to actually raise a support case ? According to the documentation here : https://docs.newrelic.com/docs/new-relic-solutions/solve-common-issues/find-help-use-support-portal/

You should be able to raise a support ticket, but I don't have a button. Can you see if you have a button and if so create a ticket directing them to this issue. Otherwise it's going to be weeks before we get a solution.

ghost commented 2 years ago

Sadly the option doesn't seem to be there for us either.

crishoj commented 2 years ago

@zsistla could you please raise to the NR team?

TheFrankman commented 2 years ago

@rjp-thijs @crishoj - Is it just the web transactions that are unknown for you ? My non-web transactions are named.

crishoj commented 2 years ago

Is it just the web transactions that are unknown for you ? My non-web transactions are named.

Same here. Only web transactions affected.

ghost commented 2 years ago

Only web transactions that fail. All non-web are named :)

raulamoretti commented 2 years ago

Laravel 8 , php 8.1, nginx and php8.1-fpm

Do you use php8.1-fpm ?

image
ZNeumann commented 2 years ago

Thank you for providing so much feedback on this issue. Unfortunately we do not yet have support for Laravel 8, though it is on the roadmap. See what frameworks are currently supported here. As a temporary workaround, disabling opcache seems to alleviate the issue. Additionally, we'd be interested in knowing if anyone is having this issue on PHP 8.0.

raulamoretti commented 2 years ago

@ZNeumann I understand, but Laravarel 8 is released in 2020-09-08!

image
raulamoretti commented 2 years ago

End of life - https://laravel.com/docs/9.x/releases

image
crishoj commented 2 years ago

Unfortunately we do not yet have support for Laravel 8

Assuming the issue can be reproduced with Laravel 7, you will commit to investigate?

crishoj commented 2 years ago

@a-james-faria @aepuls @alanwest @alexkroman @am @ZNeumann — please forgive me for this brazen mention.

Your paying customers are currently left with a dysfunctional agent for PHP, which doesn't report exceptions, and the official response so far is that support for Laravel only extends to version 7, which had end-of-life over a year ago.

Please consider devoting resources to this issue. A lack of consistent exception reporting defeats the purpose of instrumentation.

ghost commented 2 years ago

@ZNeumann As you can read in this and the other (https://github.com/newrelic/newrelic-php-agent/issues/392) thread, this happend for most people when upgrading to PHP 8.1. Same for us, on PHP 8.0 this was fine. You understand that disabling opcache isn't really a viable option as that would tank the preformance of PHP.

In before mentioned thread I tried to do a deepdive in the codebase to figure this out and it doesn't appear to be related to a change in laravel as this works some of the times, and all of the times if opcache is disabled.

TheFrankman commented 2 years ago

@ZNeumann this is disappointing response. "It's in the roadmap" offers no assurances to anyone, that could mean anything or just not be true.

To lend my voice to everyone else, NewRelic was functioning on Laravel 8.x prior to the PHP 8.1 upgrade. I am sure that you will have thousands of customers using Laravel who are soon to find out that when they upgrade to PHP8.1 the product will be broken. At this point if not supported you will have an exodus on your hands.

Support for PHP8.1 was around 3 months late and and support for Laravel8 is a year too late (laravel is actually on v9 now)

This isn't a threat but just a reality, I (and i'm sure everyone else here) simply have to have monitoring. As such I will have to look at using something else. NewRelic should also know that competitors have seen this post and I have had two different APM services reach out to me.

At the end of the day Laravel is built on Symphony, its routing is simple. Perhaps just get one of your people to have a quick look into it. Creating a VM with laravel 8.x installed with NewRelic should take your capable selves about an hour and should be part of your testing process anyway. For myself, it's probably too late, i need to get monitoring in place and i need to know it's going to be updated in reliable timeframes but you might be able to maintain your laravel customers if you act swiftly.

P.s Disabling Opcache should not even be up for discussion.

ghost commented 2 years ago

To add some more information on this I did some testing. I have a laravel 8 project on PHP7.4 with verbosedebug logging: (Working!)

2022-04-01 22:14:43.042 +0200 (462956 462956) debug: late_init called from pid=462956
2022-04-01 22:14:43.044 +0200 (462956 462956) verbosedebug: RINIT processing started
2022-04-01 22:14:43.044 +0200 (462956 462956) verbosedebug: daemon connect(fd=17 uds=@newrelic) succeeded
2022-04-01 22:14:43.044 +0200 (462956 462956) debug: added app='Local PHP Application' license='eu...AL'
2022-04-01 22:14:43.045 +0200 (462956 462956) verbosedebug: querying app='Local PHP Application' from parent=17
2022-04-01 22:14:43.045 +0200 (462956 462956) verbosedebug: sending appinfo message, len=6216
2022-04-01 22:14:43.046 +0200 (462956 462956) debug: APPINFO reply connected
2022-04-01 22:14:43.047 +0200 (462956 462956) debug: APPINFO reply full app='Local PHP Application' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg
2022-04-01 22:14:43.047 +0200 (462956 462956) debug: Adaptive sampling configuration. Connect: 1648843749000000 us. Frequency: 60000000 us. Target: 10.
2022-04-01 22:14:43.048 +0200 (462956 462956) debug: 'WT_IS_FILENAME & SCRIPT_FILENAME' naming is '/index.php'
2022-04-01 22:14:43.048 +0200 (462956 462956) verbosedebug: RINIT processing done
2022-04-01 22:14:43.057 +0200 (462956 462956) debug: detected framework = 'Laravel'
2022-04-01 22:14:43.057 +0200 (462956 462956) debug: 'Laravel' naming is 'unknown'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Foundation\Application::__construct'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Routing\RouteCollection::getRouteForMethods'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Console\Application::doRun'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Queue\Console\WorkCommand::fire'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Queue\Console\WorkCommand::handle'
2022-04-01 22:14:43.057 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Queue\Queue::createPayload'
2022-04-01 22:14:43.058 +0200 (462956 462956) debug: Laravel version is '8.40.0'
2022-04-01 22:14:43.058 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Foundation\Application::boot'
2022-04-01 22:14:43.058 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Routing\Router::prepareResponse'
2022-04-01 22:14:43.085 +0200 (462956 462956) verbosedebug: Skipping instrumented function: pid mismatch, got 462840, expected 462956
2022-04-01 22:14:43.085 +0200 (462956 462956) verbosedebug: adding custom for callable 'Illuminate\Foundation\Bootstrap\HandleExceptions::handleException'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: nr_laravel_register_after_filter: Router does not support filters
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'App\Http\Middleware\TrustProxies::handle'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'Fruitcake\Cors\HandleCors::handle'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'App\Http\Middleware\PreventRequestsDuringMaintenance::handle'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Foundation\Http\Middleware\ValidatePostSize::handle'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'App\Http\Middleware\TrimStrings::handle'
2022-04-01 22:14:43.094 +0200 (462956 462956) verbosedebug: adding custom for 'Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::handle'
2022-04-01 22:14:43.095 +0200 (462956 462956) verbosedebug: adding custom for 'App\Exceptions\Handler::render'
2022-04-01 22:14:43.095 +0200 (462956 462956) verbosedebug: adding custom for 'App\Exceptions\Handler::report'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'App\Http\Middleware\TrustProxies::handle'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'Fruitcake\Cors\HandleCors::handle'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'App\Http\Middleware\PreventRequestsDuringMainten'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'Illuminate\Foundation\Http\Middleware\ValidatePo'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'App\Http\Middleware\TrimStrings::handle'
2022-04-01 22:14:43.095 +0200 (462956 462956) debug: 'Laravel' naming is 'Illuminate\Foundation\Http\Middleware\ConvertEmp'
2022-04-01 22:14:43.098 +0200 (462956 462956) debug: nr_laravel_name_transaction: using Route::getName() for transaction naming
2022-04-01 22:14:43.098 +0200 (462956 462956) debug: 'Laravel' naming is 'login.show'
2022-04-01 22:14:43.099 +0200 (462956 462956) debug: nr_laravel_name_transaction: using Route::getName() for transaction naming
2022-04-01 22:14:43.099 +0200 (462956 462956) debug: 'Laravel' naming is 'login.show'
2022-04-01 22:14:43.099 +0200 (462956 462956) debug: txn naming freeze
2022-04-01 22:14:43.099 +0200 (462956 462956) verbosedebug: autorum: resizing buffer to 33390 bytes
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: RSHUTDOWN processing started
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: request_uri='/'
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: RSHUTDOWN processing done
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: post-deactivate processing started
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: /proc/self read: 10160 of 4096-byte pages
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: sending txnname='WebTransaction/Action/login.show' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg segment_count=59 duration=53143 threshold=2000000 priority=1.353484
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: sending transaction message, len=4148
2022-04-01 22:14:43.101 +0200 (462956 462956) verbosedebug: post-deactivate processing done

The same project under PHP8.0: (Working!)

2022-04-01 22:21:36.732 +0200 (482529 482529) debug: late_init called from pid=482529
2022-04-01 22:21:36.732 +0200 (482529 482529) verbosedebug: RINIT processing started
2022-04-01 22:21:36.733 +0200 (482529 482529) verbosedebug: daemon connect(fd=17 uds=@newrelic) succeeded
2022-04-01 22:21:36.733 +0200 (482529 482529) debug: added app='Local PHP Application' license='eu...AL'
2022-04-01 22:21:36.733 +0200 (482529 482529) verbosedebug: querying app='Local PHP Application' from parent=17
2022-04-01 22:21:36.733 +0200 (482529 482529) verbosedebug: sending appinfo message, len=6232
2022-04-01 22:21:36.733 +0200 (482529 482529) debug: APPINFO reply connected
2022-04-01 22:21:36.733 +0200 (482529 482529) debug: APPINFO reply full app='Local PHP Application' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg
2022-04-01 22:21:36.733 +0200 (482529 482529) debug: Adaptive sampling configuration. Connect: 1648843749000000 us. Frequency: 60000000 us. Target: 10.
2022-04-01 22:21:36.734 +0200 (482529 482529) debug: 'WT_IS_FILENAME & SCRIPT_FILENAME' naming is '/index.php'
2022-04-01 22:21:36.734 +0200 (482529 482529) verbosedebug: RINIT processing done
2022-04-01 22:21:36.761 +0200 (482529 482529) debug: detected framework = 'Laravel'
2022-04-01 22:21:36.761 +0200 (482529 482529) debug: 'Laravel' naming is 'unknown'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Foundation\Application::__construct'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Routing\RouteCollection::getRouteForMethods'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Console\Application::doRun'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Queue\Console\WorkCommand::fire'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Queue\Console\WorkCommand::handle'
2022-04-01 22:21:36.761 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Queue\Queue::createPayload'
2022-04-01 22:21:36.763 +0200 (482529 482529) debug: Laravel version is '8.40.0'
2022-04-01 22:21:36.763 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Foundation\Application::boot'
2022-04-01 22:21:36.763 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Routing\Router::prepareResponse'
2022-04-01 22:21:36.807 +0200 (482529 482529) verbosedebug: adding custom for callable 'Illuminate\Foundation\Bootstrap\HandleExceptions::handleException'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: nr_laravel_register_after_filter: Router does not support filters
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'App\Http\Middleware\TrustProxies::handle'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'Fruitcake\Cors\HandleCors::handle'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'App\Http\Middleware\PreventRequestsDuringMaintenance::handle'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Foundation\Http\Middleware\ValidatePostSize::handle'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'App\Http\Middleware\TrimStrings::handle'
2022-04-01 22:21:36.899 +0200 (482529 482529) verbosedebug: adding custom for 'Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull::handle'
2022-04-01 22:21:36.900 +0200 (482529 482529) verbosedebug: adding custom for 'App\Exceptions\Handler::render'
2022-04-01 22:21:36.900 +0200 (482529 482529) verbosedebug: adding custom for 'App\Exceptions\Handler::report'
2022-04-01 22:21:36.901 +0200 (482529 482529) debug: 'Laravel' naming is 'App\Http\Middleware\TrustProxies::handle'
2022-04-01 22:21:36.902 +0200 (482529 482529) debug: 'Laravel' naming is 'Fruitcake\Cors\HandleCors::handle'
2022-04-01 22:21:36.902 +0200 (482529 482529) debug: 'Laravel' naming is 'App\Http\Middleware\PreventRequestsDuringMainten'
2022-04-01 22:21:36.903 +0200 (482529 482529) debug: 'Laravel' naming is 'Illuminate\Foundation\Http\Middleware\ValidatePo'
2022-04-01 22:21:36.903 +0200 (482529 482529) debug: 'Laravel' naming is 'App\Http\Middleware\TrimStrings::handle'
2022-04-01 22:21:36.903 +0200 (482529 482529) debug: 'Laravel' naming is 'Illuminate\Foundation\Http\Middleware\ConvertEmp'
2022-04-01 22:21:36.917 +0200 (482529 482529) debug: nr_laravel_name_transaction: using Route::getName() for transaction naming
2022-04-01 22:21:36.917 +0200 (482529 482529) debug: 'Laravel' naming is 'login.show'
2022-04-01 22:21:36.919 +0200 (482529 482529) debug: nr_laravel_name_transaction: using Route::getName() for transaction naming
2022-04-01 22:21:36.919 +0200 (482529 482529) debug: 'Laravel' naming is 'login.show'
2022-04-01 22:21:36.919 +0200 (482529 482529) debug: txn naming freeze
2022-04-01 22:21:36.919 +0200 (482529 482529) verbosedebug: autorum: resizing buffer to 33391 bytes
2022-04-01 22:21:36.920 +0200 (482529 482529) verbosedebug: RSHUTDOWN processing started
2022-04-01 22:21:36.920 +0200 (482529 482529) verbosedebug: request_uri='/'
2022-04-01 22:21:36.920 +0200 (482529 482529) verbosedebug: RSHUTDOWN processing done
2022-04-01 22:21:36.920 +0200 (482529 482529) verbosedebug: post-deactivate processing started
2022-04-01 22:21:36.921 +0200 (482529 482529) verbosedebug: /proc/self read: 11512 of 4096-byte pages
2022-04-01 22:21:36.921 +0200 (482529 482529) verbosedebug: sending txnname='WebTransaction/Action/login.show' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg segment_count=286 duration=186830 threshold=2000000 priority=1.492263
2022-04-01 22:21:36.921 +0200 (482529 482529) verbosedebug: sending transaction message, len=4148
2022-04-01 22:21:36.921 +0200 (482529 482529) verbosedebug: post-deactivate processing done

And under PHP8.1: (NOT WORKING)

2022-04-01 22:26:11.668 +0200 (483027 483027) debug: late_init called from pid=483027
2022-04-01 22:26:11.669 +0200 (483027 483027) verbosedebug: RINIT processing started
2022-04-01 22:26:11.669 +0200 (483027 483027) verbosedebug: daemon connect(fd=16 uds=@newrelic) succeeded
2022-04-01 22:26:11.669 +0200 (483027 483027) debug: added app='Local PHP Application' license='eu...AL'
2022-04-01 22:26:11.669 +0200 (483027 483027) verbosedebug: querying app='Local PHP Application' from parent=16
2022-04-01 22:26:11.669 +0200 (483027 483027) verbosedebug: sending appinfo message, len=6264
2022-04-01 22:26:11.670 +0200 (483027 483027) debug: APPINFO reply connected
2022-04-01 22:26:11.670 +0200 (483027 483027) debug: APPINFO reply full app='Local PHP Application' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg
2022-04-01 22:26:11.670 +0200 (483027 483027) debug: Adaptive sampling configuration. Connect: 1648843749000000 us. Frequency: 60000000 us. Target: 10.
2022-04-01 22:26:11.670 +0200 (483027 483027) debug: 'WT_IS_FILENAME & SCRIPT_FILENAME' naming is '/index.php'
2022-04-01 22:26:11.670 +0200 (483027 483027) verbosedebug: RINIT processing done
2022-04-01 22:26:11.711 +0200 (483027 483027) verbosedebug: adding custom for callable 'Illuminate\Foundation\Bootstrap\HandleExceptions::handleException'
2022-04-01 22:26:11.762 +0200 (483027 483027) debug: txn naming freeze
2022-04-01 22:26:11.762 +0200 (483027 483027) verbosedebug: url rules: ignore=0 before='/index.php' after='/index.php'
2022-04-01 22:26:11.762 +0200 (483027 483027) verbosedebug: autorum: resizing buffer to 49144 bytes
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: RSHUTDOWN processing started
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: request_uri='/'
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: RSHUTDOWN processing done
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: post-deactivate processing started
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: /proc/self read: 10875 of 4096-byte pages
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: sending txnname='WebTransaction/Uri/index.php' agent_run_id=BS9xuMPWsjUOADSRihJaA09iR1xUAAMBAAAnIQEABwjFAgQSW3y2AwAKOS4xOS4wLjMwOQALdGhpanMtWkJvb2sAFUxvY2FsIFBIUCBBcHBsaWNhdGlvbg segment_count=1 duration=93710 threshold=2000000 priority=1.670524
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: sending transaction message, len=2564
2022-04-01 22:26:11.764 +0200 (483027 483027) verbosedebug: post-deactivate processing done

I might have broken something during testing because now all 8.1 requests end up as /index.php. All versions have had the same newrelic config!

zsistla commented 2 years ago

Hi all, we'd like to take a moment to thank you all for your civility (and even the brutal honesty) with this understandably frustrating issue.

@rjp-thijs Thanks so much for the logs!!! They were extremely helpful in guiding us to narrow down the debugging strategies.

We believe we have a fix for this issue which was related to how Laravel 8+ handles their opcache preloading with PHP 8.1. They do preloading with autoload.php and autoload_real.php even if they don't explicitly set preloading on as an ini setting and they do it slightly differently for PHP 8.1 possibly due to the opcache changes that went into PHP 8.1.

Snapshot using PHP 8.1 w/Laravel 8.83.6:

image

We're going to be testing out the fix in the coming week to ensure we covered the edge cases, then hope to get it into our upcoming release.

ghost commented 2 years ago

@zsistla Thank you for looking into this and I'm glad the logs were helpfull. Might I be so rude to ask if you can test this fix also against Laravel 9.x? 8.x gets bugfixes untill July 26th, 2022 and I'm sure more and more people will start transitioning to Laravel 9. Would be nice if this case was also tested :)

For the record: We're already using Laravel 9 in production for a while now.

zsistla commented 2 years ago

It should also work for 9.x, but we can test with that as well.

However, please see this caveat:

One caveat regarding our solution. While there was an error on our side, there does seem to be some instability with laravel and another process which is not related to NR as the following events were observed with NR disabled. If the underlying zend memory structures have been corrupted, the agent will be unable to function correctly. Some of the "unknown" issues were because we queried zend for a class and the underlying structures reported it did not exist. Once it has been corrupted, for proper functioning of the opache, it needs to be reset opcache_reset();

Laravel/server logs showed the other process would sometimes:
laravel_web    | [Mon Apr 04 13:50:59.242567 2022] [core:notice] [pid 1] AH00052: child pid 29 exit signal Segmentation fault (11)
laravel_web    | zend_mm_heap corrupted

Timeout msg:

laravel_web    | 192.168.0.1 - - [04/Apr/2022:14:25:02 +0000] "-" 408 0 "-" "-"

We believe the opcache gets corrupted at this point. After a timeout occurs, the agent is no longer able to query for functions/classes. After this, laravel performance also degrades as it appears to query the opcache, and, unable to get the correct class/function information, it loads pages directly (not via opcache). If this happens, laravel is also not using the opcache because it's become corrupted. Not sure if it is related to this: https://bugs.php.net/bug.php?id=81678 However, this portion of the issue appears to be something to bring up with the laravel and/or PHP teams.

Again, we've observed this issue with the NR agent disabled. If opcache is required, the recommendation would be to use PHP 8.0 with Laravel 8+ since the timeouts (even without the newrelic agent installed) appear to be causing the opcache to be corrupted. Alternatively, the timeout could be set to a very large number and have it reset the opcache whenever it fires.

zsistla commented 2 years ago

You can see here also, laravel is unable to query the cache:

image

It appears it tries to query the cache for the previously run URI \a but doesn't get anything so it thinks it's a null string.

ghost commented 2 years ago

I still think this is good news! I understand that there's some memory issues going on that NR has no part in. But as far as I understand it'll still work better then it currently does :) The issues you've found should also impact applications if NR is not running right? As said before we're running L9 on 8.1 now for a couple of months and haven't noticed the degration in preformance yet. Might be good to mention that we restart php-fpm fully after every deploy, which is a few times a day. Maybe this prevents this from happening? But that's not something that we need to discuss here I think.

TheFrankman commented 2 years ago

@zsistla - Any news on a possible release date ?

raulamoretti commented 2 years ago

UP, there is a date to release?

ghost commented 2 years ago

@zsistla Any update on this?

jeffcaulfield-morphsites commented 2 years ago

@zsistla This is impacting one of our largest customers since upgrading their web app to PHP 8.1 (Laravel 8) a couple weeks ago. Disabling opcache in the production environment is not a viable option. Downgrading to PHP 8.0 is highly undesirable for many reasons, including the loss of runtime performance improvements.

NewRelic has been an extremely useful tool for troubleshooting up to this point, but if this problem is not resolved soon then we'll be forced to transition the customer to another solution—we simply can't justify the monthly cost otherwise.

ghost commented 2 years ago

I just noticed a PR was opend about 12 hours ago: https://github.com/newrelic/newrelic-php-agent/pull/419