generating backpressure dd events

nirajdewani commented 2 weeks ago

Description

Now that sentry supports Datadog events, \ updating backpressure code to generate events as well (in addition to metrics).

Next steps

Send events using dogshell
Update event body if the need be

lynnagara commented 2 weeks ago

What is the motivation for this? Does it tell us anything that the metric doesn't already?

mwarkentin commented 2 weeks ago

Hey @lynnagara there was some more detail in the original PR / ticket: https://getsentry.atlassian.net/browse/OPS-5392

This would enable us to do things like:

overlay events on top of dashboard widgets
add tags to events to support analysis / grouping
Trigger DD monitors for alerting purposes (eg. notifying slack / PagerDuty)

mwarkentin commented 2 weeks ago

Here's an example time period where we experienced backpressure on May 6:

And split:

Some things that aren't clear:

Why is everything a multiple of 10?
When exactly did backpressure trigger?
When did it resolve?

Here is one ingest consumer lag graph of the same time:

With events we can have discrete events that could include information from the backpressure system for more context. These can be overlaid on top of graphs:

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 75.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.89%. Comparing base (7eb565d) to head (51ae4cd). Report is 1 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #71086 +/- ## ========================================== - Coverage 77.89% 77.89% -0.01% ========================================== Files 6525 6525 Lines 290597 290601 +4 Branches 50276 50276 ========================================== - Hits 226375 226372 -3 - Misses 57975 57979 +4 - Partials 6247 6250 +3 ``` | [Files](https://app.codecov.io/gh/getsentry/sentry/pull/71086?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) | Coverage Δ | | |---|---|---| | [src/sentry/processing/backpressure/health.py](https://app.codecov.io/gh/getsentry/sentry/pull/71086?src=pr&el=tree&filepath=src%2Fsentry%2Fprocessing%2Fbackpressure%2Fhealth.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry#diff-c3JjL3NlbnRyeS9wcm9jZXNzaW5nL2JhY2twcmVzc3VyZS9oZWFsdGgucHk=) | `83.78% <75.00%> (-0.51%)` | :arrow_down: | ... and [11 files with indirect coverage changes](https://app.codecov.io/gh/getsentry/sentry/pull/71086/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry)

getsentry / sentry