Open bobheadxi opened 3 years ago
This was brought up a few times before, but the current implementation of silencing Sourcegraph alerts are effectively permanent, where the configuration looks like:
{ "observability.silenceAlerts": [ "warning_gitserver_disk_space_remaining" ] }
Documented here: https://docs.sourcegraph.com/@3.20/admin/observability/alerting#silencing-alerts
tl;dr https://github.com/sourcegraph/sourcegraph/pull/14474#pullrequestreview-503751623
Having permanent silences is not advised as it actually a sign of a bad alert or a problem that we could be just hiding.
One possible alternative is a format that allows users to specify versions to silence the alert in:
{ "observability.silenceAlerts": [ { "name": "warning_gitserver_disk_space_remaining", "versions": [ x, y, z ] ] }
Or a mechanism to automatically expire all alerts on a version upgrade
What if we just expose the Alertmanager interface as we do with Grafana?
Heads up @davejrt @ggilmore @dan-mckean @caugustus-sourcegraph @stephanx - the "team/delivery" label was applied to this issue.
This was brought up a few times before, but the current implementation of silencing Sourcegraph alerts are effectively permanent, where the configuration looks like:
Documented here: https://docs.sourcegraph.com/@3.20/admin/observability/alerting#silencing-alerts
tl;dr https://github.com/sourcegraph/sourcegraph/pull/14474#pullrequestreview-503751623
One possible alternative is a format that allows users to specify versions to silence the alert in:
Or a mechanism to automatically expire all alerts on a version upgrade