sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.11k stars 1.27k forks source link

monitoring: silence alerts by Sourcegraph version #14503

Open bobheadxi opened 3 years ago

bobheadxi commented 3 years ago

This was brought up a few times before, but the current implementation of silencing Sourcegraph alerts are effectively permanent, where the configuration looks like:

{
  "observability.silenceAlerts": [
    "warning_gitserver_disk_space_remaining"
  ]
}

Documented here: https://docs.sourcegraph.com/@3.20/admin/observability/alerting#silencing-alerts

tl;dr https://github.com/sourcegraph/sourcegraph/pull/14474#pullrequestreview-503751623

Having permanent silences is not advised as it actually a sign of a bad alert or a problem that we could be just hiding.

One possible alternative is a format that allows users to specify versions to silence the alert in:

{
  "observability.silenceAlerts": [
    { "name": "warning_gitserver_disk_space_remaining", "versions": [ x, y, z ]
  ]
}

Or a mechanism to automatically expire all alerts on a version upgrade

pecigonzalo commented 3 years ago

What if we just expose the Alertmanager interface as we do with Grafana?

github-actions[bot] commented 2 years ago

Heads up @davejrt @ggilmore @dan-mckean @caugustus-sourcegraph @stephanx - the "team/delivery" label was applied to this issue.