search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.19k stars 4.2k forks source link

Org based rate limiter for notifications #68014

Open mrduncan opened 7 months ago

mrduncan commented 7 months ago

Incident link: https://www.notion.so/sentry/874c8e96105e425baca0bed2ba76a8a4

As a followup to a recent incident, we should implement safeguards to ensure that a single org's notifications don't backlog the notifications for all other orgs. The proposal in the linked incident doc suggests adding a rate limit on notifications we can send to a single org.

Requirements:

  1. What is the rate limit? (EM to work with Eng on this, but we can set some static ceiling based on this email task monitor, and combine that with notifications we send to other platforms (Slack, etc)
  2. What happens to notifications when they get rate limited?
  3. This feature should be turned off for self hosted (need to confirm this)
getsantry[bot] commented 7 months ago

Routing to @getsentry/product-owners-alerts for triage ⏲️

mrduncan commented 7 months ago

This draft was sitting on the old issues team board, converted to an issue since it feels relatively important/useful.