Use Laravel queue to send events to Sentry

[mnbp1371]

When I run the app('sentry')->captureException($error); , the speed of the program is very slow. i think you need to job in laravel or lumen.But this has many problems، for example: exception can not be serialize

[stayallive]

You don't give a lot of information about your setup so I'm going to make a few assumptions.

Do you have tracing enabled with traces_sample_rate set to anything higher than 0?

Otherwise we have chosen this behaviour by design. When you send an event to sentry.io we expected there to be something wrong and you want to make sure that event comes to Sentry so we can let you know, and in PHP this comes at the cost of the response time (but again most requests to your app should not have to sent anything to Sentry so it's only the requests where something is going wrong).

PHP has no way to do async stuff and we could possibly leverage the queue but that also doesn't guarantee to work if your app is having trouble so that brings a whole host of other cases where events never reach sentry.io.

We have a solution though it's a bit more involved than just flipping a switch in the config 😄

It's called Relay and it's used to relay events you send to it forward to sentry.io. This will probably solve any performance issues you are seeing. This explains it pretty well how it works (this is in the context of our performance features but works the same for captureException, it all goes through relay if you configure it that way): https://docs.sentry.io/platforms/php/performance/#improve-response-time

Because of the nature of PHP, requests in most cases are sent in the same process as you serve your users' response. In sum, this process affects the time it takes to send a request to Sentry, as it is added on top of your servers' response time. To mitigate this and reduce the addition close to zero, you can use a Relay running locally on the same machine or local network that acts as a proxy/agent. As a result, instead of your PHP process sending things to Sentry, the PHP process sends it to your local Relay, which then forwards it to Sentry.

[robsontenorio]

@stayallive how about the new ::dispatchAfterResponse() ?

https://laravel.com/docs/8.x/queues#dispatching-after-the-response-is-sent-to-browser

Even current approach with Terminable Middleware there is a noticeable lag. Please, consider ::dispatchAfterResponse() for zero lag.

[stayallive]

Hi @robsontenorio, dispatchAfterResponse is using the exact same logic as a terminable middleware so that won't help unfortunately. But I need to know a bit more since sending traces (https://sentry.io/for/performance/) to Sentry and sending exceptions to Sentry are handled differently and I don't know which one you are referring to at the moment.

Are we talking about tracing causing slower responses or are we talking about events being sent to Sentry (such as exceptions or log messages)?

What web server software are you using and are you using PHP-FPM?

[robsontenorio]

@stayallive i am talking about sending exceptions to Sentry. Yes I am using php-fpm with Nginx.

I will dig sentry package. But on my recent project just tested “dispatchAfterResponse” and worked perfectly as excepted calling an external API without lag.

[oltreseba]

To add to this, it would be great if at least the performance tracking could be done using queues.

I understand the reason behind not doing it for exception (i.e. queue may have additional problems + exception not serializable, etc), but for performance tracking it would be great.

We are tracing 0.1 of the request, and these requests are slowed down because of this. Not only this, but if for any reason, the server fails to connect to the sentry server (we have an on-premise installation, but even with sentry.io things may go bad), all these requests are slowed down until they time out.

[stayallive]

@oltreseba Although I agree, this is easier said than done unfortunately because how the PHP SDK works currently, although this can be fixed in the future of course!

We are using measures to make sure tracing does not impact the request time by using a terminable middleware which runs after the request has been sent to the user, however my assumption is that you are not using PHP-FPM or you are using Apache for which I'm not a 100% sure it works as intended.

Until we either figure out if we have a bug or are able to queue performance traces (since they might be big this probably has other challenges) running a small binary called Relay is a great way to prevent issues and ensure quick response times: https://docs.sentry.io/platforms/php/performance/#improve-response-time. So for the time being Relay might already help out, since you can run it locally on the server where the app runs requests can be sent almost instantly and Relay than works to relay it to your Sentry server or sentry.io depending on what you have setup.

[oltreseba]

I never thought it was going to be easy, I was just supporting the idea of having it eventually :).

We are using php-fpm with nginx, but we still had issues today caused by our sentry server not being reachable, this caused a lot of php-fpm processes to hang waiting for a reply, and eventually, lead to the saturation of the available processes, and request being dropped/timing out, etc. Queues may have helped in this situation.

I will check Relay, thank you for the input.

[stayallive]

Ah okay, yeah for when Sentry is unavailable that will not help since the FPM process is still processing, but initially the user should not have seen that, but the process exhaustion is noticeable of course. I would recommend Relay either way even if we have queueing later 😄

[iker-barriocanal]

This issue looks like a question that has already been answered, so closing it. Feel free to comment here if you have further questions or think we should work on this.

Thanks!

[stayallive]

@iker-barriocanal sorry that is my bad, this should've been tagged as a feature request! Fixed now!

[joriswvanrijn]

Is there any progress on this feature request? Would love to do the same!

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[ghost]

Is there any progress on this?

[cong08]

I have the same problem

[loranger]

Would love a queued error tracking too, in order to speed up error rendering

[robsontenorio]

@stayallive i am talking about sending exceptions to Sentry. Yes I am using php-fpm with Nginx.

I will dig sentry package. But on my recent project just tested “dispatchAfterResponse” and worked perfectly as excepted calling an external API without lag.

@stayallive Is that a not valid approach to improve Sentry sending exceptions?

[d8vjork]

Now that Sentry is suffering from an outage, we're having big problems on our application response time

For these cases, a PR with this makes much more sense

[zhaozhenxiang]

Hi. I also need to put the send request in the queue for execution.

[zhaozhenxiang]

Hi. I also need to put the send request in the queue for execution.

This is my temporary solution.

[ghost]

Would love an update on this.

[cleptric]

While I agree that, at least in the context of performance monitoring, utilizing a queueing system might be a great idea to reduce overhead, we also have to take applications into account that do not use such queues.

Sentry already provides a great solution to minimize the mentioned performance overhead, which we call a local Relay (sidecar). https://docs.sentry.io/platforms/php/guides/laravel/performance/#improve-response-time

The binary has no dependencies and should work on almost every OS you might use in production. You can either deploy it side by side with your app or run a centralized instance in your k8s/container environment. https://github.com/getsentry/relay

Using a local Relay will drastically reduce response time when Sentry is involved, so you should give it a try! Please let us know if you run into any issues.

[stayallive]

I also want to add that we released a update to the underlying PHP SDK a few months ago that disables retries by default and in other releases we decreased the timeouts so if it happens that the Sentry instance you are using is down the impact is far less extreme: https://github.com/getsentry/sentry-php/releases/tag/3.5.0.

Still Relay is a really good product to have where you are in control over it and Relay hapilly accepts events even though the Sentry instance it's relaying to is down.

[d8vjork]

So... my only concern about this Relay thing... would this prevent my app to suffer from Sentry platform outages? If so would it retry the events by itself so my PHP app doesn't need to lose data events sent?

Will definitely give a try on our infrastructure

[antonpirker]

Yes, relay does retry in case the data can not be sent on the first try. See also here for all the configuration you can set in relay: https://docs.sentry.io/product/relay/options/

[cleptric]

I'll close this, as we are offering a generic solution that works for all PHP folks.

[robsontenorio]

But what is the solution ?

[cleptric]

@robsontenorio https://github.com/getsentry/sentry-laravel/issues/452#issuecomment-1237919922

Using a local Relay https://docs.sentry.io/product/relay/

[summerKK]

see https://github.com/getsentry/sentry-laravel/issues/792#issuecomment-1815988621