search

mperham / inspeqtor

Monitor your application infrastructure!
GNU General Public License v3.0
1.66k stars 72 forks source link

Emails should be per-cycle, not per-rule #24

Open mperham opened 9 years ago

mperham commented 9 years ago

When something breaks, it's often the case that several rules will break at the same time. Each cycle's email should summarize all the rules that are broken and not a single email per rule. This summary allows the reader to see everything broken at a glance and possibly understand the root cause.

felixbuenemann commented 9 years ago

That's a great idea!

felixbuenemann commented 9 years ago

I've had a case were redis didn't properly restart after a package upgrade and consequently a lot of other services failed and I was bombarded with thousands of emails by Inspeqtor, so some kind of throttling/digest would be a great idea, especially because process id changes always cause alerts and can very quickly add up. I could probably mitigate the spamming by setting respawn limits on the upstart services, but it'll still generate a burst of messages until those kick in.

Maybe if emails would be per cycle digests, there could be an extra global setting to only send mails every n cycles for a simple form of throttling.

nick-desteffen commented 7 years ago

This is a great idea. We use Inspeqtor for monitoring and restarting our Sidekiq processes. Whenever the memory threshold is reached we get 4 emails, ☹️ . During traffic spikes our inboxes are flooded.