search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.02k stars 175 forks source link

Allow silencing based on event labels #4146

Open raags opened 3 years ago

raags commented 3 years ago

Feature Suggestion

Right now silences can only be created for an entity or subscription. This maps well for checks that are run by sensu backend, but events can also be created ad-hoc by using the sensu agent API. These can add arbitrary entities/proxy entities. For silencing these entities, labels provide more flexibility.

Possible Implementation

Extend the silence API to allow silencing by entity and/or event labels

Context

An application sends alerts/events via sensu agent event API. These events have multiple labels, for e.g. component, team, etc. Silencing a group of these events it not possible now.

portertech commented 3 years ago

This is interesting, good food for thought. We've started looking into what it would take to implement this. Silencing currently relies on a combination of subscription and check name for its resource name (key in the store). We like the idea of silencing with labels, but more investigation is required before we can commit to an action.

raags commented 3 years ago

Another use case this maps to is for Grafana Alerts. Right now there is no working sensugo plugin in Grafana. One of the difficulties is in modelling the Grafana alert in Sensu since each time series can be considered as a separate alert. This can be done, but then silencing does not work.

Instead, we are now currently using Alertmanager specifically for this. With label based silencing + the Grafana plugin, I can move completely to sensugo for all my alert routing needs.

calebhailey commented 3 years ago

Great point RE: Grafana alerts!

A PR to add support for Sensu Go notifications has been merged and appears to be scheduled for the Grafana 7.4.0 release; see: https://github.com/grafana/grafana/pull/28012

Grafana notifications are basically converted to events, sent via POST /api/core/v2/namespaces/:namespace/events/:entity/:check. Sensu will automatically create the corresponding entity for this event if one does not already exist, but by default that entity will not have any subscriptions added (IIRC), including the entity: subscription. These entities also won't have any labels added by default.

If we decide we can't add Silencing by label at this time, ensuring that proxy entities created via the API have the entity: subscription for silencing purposes might be a viable workaround.

/cc @portertech

betorvs commented 3 years ago

any news about this @portertech ?