search

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
https://develop.sentry.dev/self-hosted/
Other
7.76k stars 1.75k forks source link

Event data was getting dropped in relay #3335

Open sree-warrier opened 4 days ago

sree-warrier commented 4 days ago

Self-Hosted Version

23.11.2

CPU Architecture

x86_64

Docker Version

na

Docker Compose Version

na

Steps to Reproduce

Events were getting dropped at the relay side also we were facing issue with envelope buffer size limit. Is there any setting at relay side to controlling the event size and payload under a event ?

How we can check the payload of a event coming to sentry, does we get a accumulated value of common errors with the size of a event with payload. We know that event size depends on the breadcrumbs and dependencies and other configuration setting of app.

How can restrict the size of data at relay side if an event payload i having more than a limit.

Expected Result

NA

Actual Result

NA

Event ID

NA

aldy505 commented 4 days ago

The default envelope size is 50 MB (see https://docs.sentry.io/product/relay/options/#size-limits, config key is limits.max_envelope_size), this should be restricted on each SDK. The maximum buffer size for envelope is 1000 (config key is cache.envelope_buffer_size), so in total it's 1000 * 50 MB (= 50 GB stored in memory). But you might also want to look at the spooling config (https://docs.sentry.io/product/relay/options/#spooling)

Usually the problem is caused by either:

  1. relay having a hard time publishing that events to Kafka. Probably slow disk speed or low RAM.
  2. The event throughput is too high, if this is the case, you can spawn a few replica of the relay, but bear in mind you'll also need to increase your server (or VM) specs.
sree-warrier commented 1 day ago

@aldy505 is there a way in UI or metric API to check per event payload size ?