[grafana] How to import alerts from dashboard

[Mattie112]

If a am correct it should now be possible to add/imports alerts just like dashboards since https://github.com/grafana/helm-charts/pull/1847?

I am using https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack which in the end uses this chart. I have my config set to:

  sidecar:
    dashboards:
      enabled: true
      searchNamespace: ALL
    alerts:
      enabled: true
      searchNamespace: ALL

I have a dashboard with an alert to trigger when < 7000 in query A:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 7,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "alert": {
        "alertRuleTags": {},
        "conditions": [
          {
            "evaluator": {
              "params": [
                7000
              ],
              "type": "lt"
            },
            "operator": {
              "type": "and"
            },
            "query": {
              "params": [
                "A",
                "5m",
                "now"
              ]
            },
            "reducer": {
              "params": [],
              "type": "avg"
            },
            "type": "query"
          }
        ],
        "executionErrorState": "alerting",
        "for": "5m",
        "frequency": "1m",
        "handler": 1,
        "message": "If this alert triggers[..]",
        "name": "xx xx alert",
        "noDataState": "no_data",
        "notifications": []
      },
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "decimals": 0,
      "fieldConfig": {
        "defaults": {
          "links": []
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 9,
        "w": 7,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 9,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "max": true,
        "min": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "9.2.4",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "expr": "xx{type=\"xx\"}",
          "legendFormat": "{{type}}",
          "refId": "A"
        }
      ],
      "thresholds": [
        {
          "colorMode": "critical",
          "fill": true,
          "line": true,
          "op": "lt",
          "value": 7000,
          "visible": true
        }
      ],
      "timeRegions": [],
      "title": "xx xx",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "transparent": true,
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:621",
          "decimals": 0,
          "format": "none",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:622",
          "format": "none",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
  [...]
  ],
  "refresh": false,
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Nodes",
  "uid": "xx",
  "version": 1,
  "weekStart": ""
}

And this is present in a CM:

apiVersion: v1
data:
  grafana-xx-xx-xx.json: |
    {..json above..}
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    grafana_dashboard: "1"
    grafana_alert: "1"
  name: grafana-dashboard

The dashboard works fine, I can see the graph. But: I don't see the alert. I was expecting it to show up at the alerts tab.

The grafana pod has a volume for both alerts and dashboard

Log from the grafana-sc-alerts

{"time": "2022-11-18T13:08:48.882609+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2022-11-18T13:08:48.882862+00:00", "level": "WARNING", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2022-11-18T13:08:48.883163+00:00", "level": "INFO", "msg": "Loading incluster config ..."}
{"time": "2022-11-18T13:08:48.884195+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://172.20.0.1:443' loaded..."}
{"time": "2022-11-18T13:08:48.884354+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2022-11-18T13:08:48.884513+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}

What am I missing here? Why does it not see my alert?

[zanhsieh]

@Dima-Diachenko-work Do you have any idea on this ^^^ ?

[ghost]

I'm also interested in this, but I don't know yet. I didn't try this functionality, but I will do it in near few weeks. I will share it If I have the result. But maybe someone else already realized how to do it?

[Mattie112]

@jleloup sorry for tagging you but since you created the MR: can you provide some more info on how this should work?

[diego-zurita-bayas]

We also are having a struggle to add datasources: sidecar: alerts: enabled: true labelValue: "1" dashboards: enabled: true labelValue: "1" datasources: enabled: true labelValue: "1" searchNamespace: ALL notifiers: enabled: true labelValue: "1" plugins: enabled: true labelValue: "1"

and it doesn't work removing either the label labelValue: "1"

[Mattie112]

Any updates on this? We are still struggling with this issue on how to configure alerts.

[jwreford99]

Hi @Mattie112 , if I am understanding you correctly, you seem to be including alerts in the dashboard description itself. I think (and I'm no expert) that since v9 alerts got separated to be their own distinct object, rather than being included alongside the dashboard.

The JSON you posted looks very similar to how our JSON used to look (and that worked fine until we upgraded to v9 of Grafana). I think since that upgrade the alert rules are provisioned completely separately (described a bit more here). So they aren't embedded in the dashboard description, but instead link to the panel/dashboard

I have got this sidecar working, by producing a different YAML file for alerts.

I have

sidecar:
  alerts:
    enabled: true
    label: grafana_alert

on the helm file which is deployed in the monitoring namespace

Then in my cluster I create configmaps that are a bit like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: alert-julian
  namespace: monitoring
  labels:
    grafana_alert: "1"
data:
  provision-test.yml: |-
    apiVersion: 1
    groups:
        - orgId: 1
          name: provision-test
          folder: Julian
          interval: 1m
....

The outer YAML is simply a configmap with an annotation which tells the sidecar that it is an alert (note it is also in the same namespace as the sidecar).

The inner YAML is then a file, which the sidecar will pick up and copy into Grafana's filesystem.

I got the inner YAML by calling curl --insecure -H "Authorization: Bearer [SERVICE ACCOUNT TOKEN]" {grafanaUrl}/api/v1/provisioning/alert-rules/export which returns all of the alerts already made in Grafana and I made the configmaps from there.

Does that make sense? (as I said, I'm not an expert on any of this, but this is how I got it working for us)