search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.14k stars 4.2k forks source link

Filter Alerts by [time of day] and [day of week] #70310

Open mark-langer opened 6 months ago

mark-langer commented 6 months ago

Problem Statement

We'd like to get informed on slow average query times, but only in certain time ranges.

We know from our previous APM tool that many alerts came in unnecessarily because the average response time of our servers was very high because of very few requests at night. E.g. a single customer generating two documents at 3 am in the morning drove up the average time altough it wasn't overburdening our servers; it just takes 10-15 seconds to generate each one.

Solution Brainstorm

We'd like to filter alerts so that they are only triggered if...

Product Area

Alerts

getsantry[bot] commented 6 months ago

Assigning to @getsentry/support for routing ⏲️

getsantry[bot] commented 6 months ago

Routing to @getsentry/product-owners-alerts for triage ⏲️

Dhrumil-Sentry commented 6 months ago

@mark-langer thanks a lot for this feedback. It seems like you basically want to get rid of daily/weekly seasonality based false positives from these alerts i.e. Sentry should be smart enough to see a drop and know this drop happens every evening.

Is that accurate?

mark-langer commented 6 months ago

Yes!

On Tue, May 7, 2024 at 22:59 Dhrumil @.***> wrote:

@mark-langer https://github.com/mark-langer thanks a lot for this feedback. It seems like you basically want to get rid of daily/weekly seasonality based false positives from these alerts i.e. Sentry should be smart enough to see a drop and know this drop happens every evening.

Is that accurate?

— Reply to this email directly, view it on GitHub https://github.com/getsentry/sentry/issues/70310#issuecomment-2099300358, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKNHIVULBNZH6DTSMEHFZLZBE6BXAVCNFSM6AAAAABHIT3V76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJZGMYDAMZVHA . You are receiving this because you were mentioned.Message ID: @.***>

Dhrumil-Sentry commented 6 months ago

We're considering building anomaly detection-based alerts in Sentry. Could you please share what type of metrics you get alerted on beyond response times?

Also do you setup alerting for specific endpoints or response times for all your APIs/pages?