Failsafe system if Sentry goes down

[thierroboto]

Problem Statement

As you know, Sentry event ingestion went down a few times in the past week. When that happens, our website goes down with it, as all the requests start timing out.

Some kind of failsafe system that would deactivate sentry temporarily in these cases would be very appreciated.

Solution Brainstorm

Perhaps a query to the Sentry status page to see if everything is operational before trying to send the events to Sentry?

[stayallive]

Which version of Sentry are you using?

With the current timeouts in place it shouldn't happen that your site is affected unless you are sending MANY events to Sentry in a single request. Also performance transactions are sent after the response is sent to the user so that also shouldn't affect it. Not saying it wasn't but curious to learn more about your environment to figure out if there are things we could improve still to prevent affecting applications when Sentry has downtime.

---

I do also want to mention that the best way to prevent Sentry from affecting your application is running Relay, in the docs it is mentioned as a way to improve response times (which it also does) but as an added benefit if Sentry is down your Relay instance is not and will accept events and even queue them until they can be forwarded to Sentry. See: https://docs.sentry.io/platforms/php/performance/#improve-response-time.

[thierroboto]

Thanks for the quick reply. We're using the following: "sentry/sdk": "^3.4", "sentry/sentry": "^3.20.1", Should I update?

I am unsure if i'm sending many requests at the same time, but i just noticed my sample rate was at 1.0. I just brought it down to 0.2, will that help in the meantime?

I'm looking into the relay right now. We're using both heroku and AWS to host our applications, which would you suggest to deploy relay? if AWS, which service?

Thank-you

[stayallive]

Those versions are new enough to include the latest timeout changes. So no need to upgrade 👍

Just to clarify, you are using the Laravel SDK right? Or are you using just the PHP SDK?

[thierroboto]

The laravel sdk

[cleptric]

I do not recommend lowering the sample_rate. This could cause you to miss out on important errors. A local Relay can be hosted on any x86 Server/VM, so either ECS using our docker image or an EC2 instance running the binary directly.

I don't have much experience with Heroku though, so I can't give you any recommendations for it.

[summerKK]

Hello, I found that `sentry` can be reported asynchronously through some configurations.

My `sentry-laravel` version is `3.8.2` Some files need to be created. like this: Our main purpose is to replace the original `HttpClient` (sync), with `QueueClient` (async). TransportFactory.php ```php requestFactory = Psr17FactoryDiscovery::findRequestFactory(); $this->streamFactory = Psr17FactoryDiscovery::findStreamFactory(); $this->logger = $logger; $this->httpClientFactory = new HttpClientFactory( null, null, $this->streamFactory, // Reporting using queues new QueueClient($options), Version::SDK_IDENTIFIER, Version::SDK_VERSION ); } /** * {@inheritdoc} */ public function create(Options $options): TransportInterface { if (null === $options->getDsn()) { return new NullTransport(); } return new HttpTransport( $options, $this->httpClientFactory->create($options), $this->streamFactory, $this->requestFactory, new PayloadSerializer($options), $this->logger ); } } ``` ReporterJob.php ```php method = $method; $this->url = $url; $this->options = $options; } /** * Execute the job. */ public function handle() { try { $httpClient = HttpClient::create(); $this->options['body'] = base64_decode($this->options['body']); $response = $httpClient->request($this->method, $this->url, $this->options); } catch (Throwable $e) { } } } ``` QueueClient.php ```php options = $options; } public function sendAsyncRequest(RequestInterface $request): FulfilledPromise { $body = $request->getBody(); if ($body->isSeekable()) { $body->seek(0); } $options = [ 'headers' => $request->getHeaders(), 'body' => base64_encode($body->getContents()), 'buffer' => true, ]; if ('1.0' === $request->getProtocolVersion()) { $options['http_version'] = '1.0'; } $reporterJob = new ReporterJob($request->getMethod(), (string)$request->getUri(), $options); dispatch($reporterJob)->onQueue('tracing'); return new FulfilledPromise(new MockResponse('{"id":"1"}')); } } ``` Sentry.php ```php bound(HubInterface::class)) { $clientBuilder = app()->make(ClientBuilderInterface::class); $transport = (new TransportFactory($clientBuilder->getOptions()))->create($clientBuilder->getOptions()); $hub = app(HubInterface::class); $client = new Client($clientBuilder->getOptions(), $transport, Version::SDK_IDENTIFIER, Version::SDK_VERSION); $hub->bindClient($client); } } } ``` Finally, initialize the `AppServiceProvider` and you're done! ```php class AppServiceProvider extends ServiceProvider { /** * Bootstrap any application services. */ public function boot() { ... Sentry::init(); } ``` I hope it will help you, I hope the official can also provide some configurations to choose synchronous and asynchronous, so you don't have to be so complicated😂😂😂

[summerKK]

The 4.0 version makes it easier to send using queues. Just a simple modification is needed to seamlessly support it.

## step0: Create a `ReporterJob` ```php stringBody = $stringBody; } /** * Execute the job. */ public function handle() { try { $httpClient = new HttpClient(Version::SDK_IDENTIFIER, Version::SDK_VERSION); $request = new Request(); $request->setStringBody($this->stringBody); $response = $httpClient->sendRequest($request, $this->getOption()); } catch (Throwable $e) { // } } protected function getOption(): Options { $config = config('sentry'); foreach (static::LARAVEL_SPECIFIC_OPTIONS as $laravelSpecificOptionName) { unset($config[$laravelSpecificOptionName]); } return new Options($config); } } ``` ## step1: Create a `QueueClient` ```php getStringBody()); Queue::tracing($reporterJob); // $reporterJob->handle(); // dispatch($reporterJob)->onQueue('lv.tracing-sentry'); return new Response(200, [], ''); } /** * This method will be invoked when using `config:cache`. An error will occur if this method is not present. */ public static function __set_state(array $state_array): self { $object = new static(); foreach ($state_array as $prop => $state) { if (! property_exists($object, $prop)) { continue; } $object->{$prop} = $state; } return $object; } } ``` ## Finally, add the configuration to `sentry.php`. ```php 'http_client' => new QueueClient(), ```

[cleptric]

Our official recommendation is to use a local Relay https://docs.sentry.io/platforms/php/guides/laravel/performance/.