search

DRuggeri / alertmanager_gotify_bridge

A bridge between Prometheus AlertManager and a Gotify server
Apache License 2.0
65 stars 9 forks source link

readme

Alertmanager to Gotify webhook bridge

An Alertmanager-aware webhook endpoint that converts messages to Gotify API calls.

Typically, this should have been created as a Gotify Plugin... but after much trial and error, the plugin system never could build properly. I also prefer the idea of separating the translation layer from needing to be built on the exact version of the running Gotify server.

Installation

Binaries

Download the already existing binaries for your platform:

$ ./alertmanager_gotify_bridge <flags>

From source

Using the standard go install (you must have Go already installed in your local machine):

$ go install github.com/DRuggeri/alertmanager_gotify_bridge
$ alertmanager_gotify_bridge <flags>

Usage

NOTE: All parameters may be set as environment entries as well as provided on the command line. The environment entry is the same as the flag but converted to all capital letters.

For example, the environment entry for gotify_token may be set as GOTIFY_TOKEN

Flags

usage: alertmanager_gotify_bridge [<flags>]

Flags:
  --help                        Show context-sensitive help (also try --help-long and --help-man).
  --gotify_endpoint="http://127.0.0.1:80/message"
                                Full path to the Gotify message endpoint ($GOTIFY_ENDPOINT)
  --bind_address=0.0.0.0        The address the bridge will listen on ($BIND_ADDRESS)
  --port=8080                   The port the bridge will listen on ($PORT)
  --webhook_path="/gotify_webhook"
                                The URL path to handle requests on ($WEBHOOK_PATH)
  --timeout=5s                  The number of seconds to wait when connecting to gotify ($TIMEOUT)
  --title_annotation="summary"  Annotation holding the title of the alert ($TITLE_ANNOTATION)
  --message_annotation="description"
                                Annotation holding the alert message ($MESSAGE_ANNOTATION)
  --priority_annotation="priority"
                                Annotation holding the priority of the alert ($PRIORITY_ANNOTATION)
  --default_priority=5          Annotation holding the priority of the alert ($DEFAULT_PRIORITY)
  --metrics_auth_username=METRICS_AUTH_USERNAME
                                Username for metrics interface basic auth ($AUTH_USERNAME and $AUTH_PASSWORD)
  --metrics_namespace="alertmanager_gotify_bridge"
                                Metrics Namespace ($METRICS_NAMESPACE)
  --metrics_path="/metrics"     Path under which to expose metrics for the bridge ($METRICS_PATH)
  --extended_details            When enabled, alerts are presented in HTML format and include colorized status (FIR|RES), alert start time, and a link to the generator of the alert ($EXTENDED_DETAILS)
  --dispatch_errors             When enabled, alerts will be tried to dispatch with a error-message regarding faulty templating or missing fields to help debugging ($DISPATCH_ERRORS)
  --debug                       Enable debug output of the server
  --version                     Show application version.

Token Override

By default, the bridge sends alerts to the initialized bridge Gotify token. This configuration allows all alerts from alertmanager to send to a single Gotify application based on the token.

The bridge supports overriding the initialized bridge Gotify token by setting the token query string parameter, which allows different receivers to send alerts to other applications in Gotify.

CURL Example:

curl http://127.0.0.1:8080/gotify_webhook?token=GS46-fGs.gW-gE. -d '
{ "alerts": [
  {
    "annotations": {
      "description":"A description",
      "summary":"A summary",
      "priority":"critical"
    },
    "status": "firing",
    "generatorURL": "http://foobar.com"
  }
]}
'

YAML Example:

receivers:
- name: storage
  webhook_configs:
  - url: http://127.0.0.1:8080/gotify_webhook?token=GS46-fGs.gW-gE.
    send_resolved: false

Templating

The supports Go templating with Prometheus-enhanced functions, so you can customize the alert messages further with templates in the title and message annotations.

For example add following line to the title:
{{if eq .Status "firing"}}🔥{{else}}✅{{end}}
This differentiates firing from resolving alerts.

Also, there are two methods you can use for additional customisation:

.Values                 Access alert-values, -labels and -metrics. 
                        Returns list of:
                            Metric  string
                            Labels  map[string]string
                            Value   float64

.Humanize <float64>     Rounds float and stripps trailing zeros to return more readable float.
                        .Humanize 5.3234134 returns 5.32
                        .Humanize 5.0       returns 5

To give further information and examples for use-cases for these methods: Imagine a simple uptime-metric for multiple instances or jobs. If you configure an alert, it would fire if any instance or alert is down. The message would probably say something like "an instance or job is down". But from the message you would not know which of the jobs or instances is the down one, or if there are multiple. To address this you have to use the .Values method. A alert-description could look like this:

{{if eq .Status "firing"}}
Following Providers are down: 
{{range $i, $provider := .Values}}
{{$provider.Labels.job}}, 
{{end}}
{{else}}
All providers are back up
{{end}}

Now if the alert fires it would list the jobs that are down. Which information the .Values method contains can be inspected in the Grafana alertmanager when configuring an alert and clicking the Preview Alert button.

Template Functions

The bridge uses a subset of Prometheus's template functions. Some of the template functions are not supported in the bridge. The file prometheus_template_functions.go contains the list of functions and how they are implemented in the bridge.

Example:

{{ reReplaceAll ".+\\|" " " .Labels.log }}

CURL Example1:

curl http://127.0.0.1:8080/gotify_webhook -d '
{ "alerts": [
  {
    "annotations": {
      "description": "{{ match \"my+\" \"my|test\" }}",
      "summary":"A summary",
      "priority":"critical"
    },
    "status": "firing",
    "generatorURL": "http://foobar.com"
  }
]}
'

CURL Example2:

curl http://127.0.0.1:8080/gotify_webhook -d '
{ "alerts": [
  {
    "annotations": {
      "description": "{{ reReplaceAll \"fir\" \" \" .Status }}",
      "summary":"A summary",
      "priority":"critical"
    },
    "status": "firing",
    "generatorURL": "http://foobar.com"
  }
]}
'

Bridge Message Templating

The bridge supports user-defined templating for all inbound messages. The user-defined templating can be used for the title and/or message. Also, user-defined templating overrides the default title and message annotations. All keys and values in the JSON from alertmanager can be used in the user-defined template. Any failures in the templates will result in the bridge falling back to default alerting.

Usage Notes:

Usage Hints:

Usage Example:

This example will show the creation of five different user-defined templates for more advanced match routing within the bridge. The example below will display a message being received and how to create a custom template based on that message.

JSON Discover Hint: Enable debugging on the bridge to see the JSON output sent from the sender to add additional values to your template.

  1. Write down the Gotify software token used in the alert message. 
    GS46-fGs.gW-gE.
  2. Find the incoming JSON for the user-defined alert. 
    { "alerts": [
  {
    "annotations": {
      "description": "{{ humanize 1234567.0 }}",
      "summary":"A summary",
      "priority":"critical"
    },
    "status": "firing",
    "generatorURL": "http://foobar.com",
  }
]}
  3. Create a file in the ./templates folder called app_message_template.tmpl.
  4. Past the following into the file from step 3. 
    {{ define "GS46-fGs.gW-gE." }}
{{ if eq (.Status) ("firing") }} {{ template "my_app1" .}} {{end}}
{{ if eq (.Annotations.summary) ("A summary") }} {{ template "my_app2" .}} {{end}}
{{ if eq (.GeneratorURL) ("http://foobar.com") }} {{ template "my_app3" .}} {{else}}Testing Message: None Found {{end}}
{{end}}
  5. Create a file in the ./templates folder called app1_message_sub-template.tmpl.
  6. Past the following into the file from step 4. 
    {{ define "my_app1" }}
{{ if eq (.Status) ("firing") }}App1 Firing {{else}}App1 Not Firing {{end}}
{{end}}
  7. Create a file in the ./templates folder called app2_message_sub-template.tmpl.
  8. Past the following into the file from step 5. 
    {{ define "my_app2" }}
{{ if eq (.Status) ("firing") }}App2 Firing {{else}}App2 Not Firing {{end}}
{{end}}
  9. Create a file in the ./templates folder called app3_message_sub-template.tmpl.
  10. Past the following into the file from step 8. 
    {{ define "my_app3" }}
{{ if eq (.Status) ("firing") }}App3 Firing {{else}}App3 Not Firing {{end}}
{{end}}
  11. Create a file in the ./templates folder called app_title_template.tmpl.
  12. Past the following into the file from step 8. 
    {{ define "title=AFRy-fWs.jW-gE." }}
{{ if eq (.Status) ("firing") }}Title Sample {{else}}Nope {{end}}
{{end}}

  13. Send the alert to the bridge, and the result will be listed below in text format.

    • Note: This example is a simple example using the same incoming JSON, but this can support different JSON being sent through the same token, but shows you can get creative with your templates. Result: 
      
Title Sample

    App1 Firing

    App2 Firing

    App3 Firing

Metrics

The bridge tracks telemetry data for metrics within the server as well as exposes gotify's health (obtained via the /health endpoint) as prometheus metrics. Therefore, the bridge can be scraped with Prometheus on /metrics to obtain these metrics.

Exported metrics:

Docker

An official scratch-based Docker image is built with every tag and pushed to DockerHub and ghcr. Additionally, PRs will be tested by GitHubs actions.

The following images are available for use:

Docker-Compose

  alertmanager_gotify_bridge:
    image: ghcr.io/druggeri/alertmanager_gotify_bridge:master
    container_name: alertmanager_gotify_bridge
    environment:
      - GOTIFY_TOKEN=xxxxxxx
      - GOTIFY_ENDPOINT=http://gotify:80/message
    volumes:
      - <local path>:/./templates
    ports:
      - 8080:8080
    restart: unless-stopped

Supported tags:

Optional bind mount:

Community Contributions

Testing and Development

If you would like to experiment and test your bridge configuration, you can simulate Prometheus alerts like so

Start the bridge with your choice of parameters. For example: set a bogus token, enable debug, and listen on port 8080 while accepting all other defaults: GOTIFY_TOKEN=FOO ./alertmanager_gotify_bridge --port=8080 --debug

You can then send a request with cURL to see if your configuration works out as expected:

curl http://127.0.0.1:8080/gotify_webhook -d '
{ "alerts": [
  {
    "annotations": {
      "description":"A description",
      "summary":"A summary",
      "priority":"critical"
    },
    "status": "firing",
    "generatorURL": "http://foobar.com"
  }
]}
'