getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
39.23k stars 4.21k forks source link

ref(crons): Include clock-ticks in volume recording in serial mode #81016

Closed evanpurkhiser closed 2 days ago

evanpurkhiser commented 2 days ago

This doesn't actually affect production sentry.io and is more of a consistency change.

Right now the batched consumer processor will record clock tick messages as volume, which is actually likely what we want to happen. In the case that ALL up-stream ingestion stops, we want to still have SOME volume for each tick so that we can detect the drop-off and start an incident.

evanpurkhiser commented 2 days ago

Yeah just so it's never zero. If it's zero the clock tick decision will just end up Normal.

I am also considering changing the code so if it's zero it explicitly computes a -100% deviation. But I kind of prefer this, since in order for the clock to even tick we HAVE to have something in the topic, so if it ticks and has no volume, something else could be wrong and I would rather it not fail closed