Configurable Alerting Definition

[epociask]

Problem

Currently, the severity routing configuration is hardcoded. While this works for some opinionated use cases, it fails to be generalized to many unique use cases. Additionally the service only supports integration with a single slack webhook and two pagerduty integration keys. Ideally, a user could have up to many slack webhooks and pager-duty services to alert.

Problem Solution

Support some alerting definition JSON that allows for severity routing definitions of multiple downstream dependencies for a severity. This will be a global definition that the application ingests and uses to understand where to alert.

This would look something like:

{
    "sev_low": [
          {
            "destination": "slack",
            "config": {},
          }
       ]
     "sev_mid": [
         {
           "destination": "slack",
           "config": {},
          },{
              "destination": "pagerduty",
              "config": {},
         }]
}

[adrain-cb]

I think this config structure makes a lot of sense, but you could argue a yaml file might be more readable for users. I think the genesis file makes sense to be json because it mimics the api payload for spawning new heuristics/pipelines but in the instance of alert configuration yaml might be more suitable for ease of use. Could look something like this:

sev_low:
- destination: slack
  config:
    url: url
    channel: "#channel"
sev_mid:
- destination: slack
  config:
    url: url
    channel: "#channel"
- destination: pagerduty
  config:
    integration_key: key
    url: url

This structure would allow for comments within the file and easy addition of different channels via environment variable loading.

[epociask]

That makes sense to me! Only slight consideration is the double management of external file types long-term and ensuring that we can easily integrate with existing docker flows.