Prometheus: variables in adhoc filters not working since 10.2 version

[cazeaux]

What happened?

When you use a variable as a value for an adhoc filter, the variable is not evaluated and the filter is applied with the variable name. The problem is seen with Prometheus datasource.

Problem seen first in 10.2.1, and confirmed in 10.4.2

What did you expect to happen?

The variable should be evaluated, and the filter applied with the variable value instead of its name.

Did this work before?

It was working at least up to version 10.1.2.

How do we reproduce it?

1. Create a query variable, let's call it $platform. For example, a variable which provides values of a given label of a metric. For the example we use: `$platform = label_values(up, platform)
2. Create an adhoc variable, and in this variable apply the filter : platform=~$platform
3. In 10.1.2, the panels are correctly filtered with the values of $platform. See inspect below:

5. In 10.2+, the panels are filtered with platform=~$platform. See inspect below:

An example dashboard to reproduce the bug (you might change platform label to another label):

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 1012,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "paasquerier"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "paasquerier"
          },
          "editorMode": "code",
          "expr": "sum(up) by (platform)",
          "instant": false,
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Panel Title",
      "type": "timeseries"
    }
  ],
  "refresh": "",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": [
            "qp",
            "testbed",
            "tb-sd"
          ],
          "value": [
            "qp",
            "testbed",
            "tb-sd"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "paasquerier"
        },
        "definition": "label_values(up,platform)",
        "hide": 0,
        "includeAll": true,
        "multi": true,
        "name": "platform",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(up,platform)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "paasquerier"
        },
        "filters": [
          {
            "key": "platform",
            "operator": "=~",
            "value": "$platform"
          }
        ],
        "hide": 0,
        "name": "platform_filter",
        "skipUrlSync": false,
        "type": "adhoc"
      }
    ]
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Grafana Bug",
  "uid": "ddlp24t4jkm4gd",
  "version": 11,
  "weekStart": ""
}

Is the bug inside a dashboard panel?

No

Environment (with versions)?

Grafana: 10.2.1 and recent versions OS: Windows 10 Browser: Edge, Vivaldi, Chrome, Firefox

Grafana platform?

Kubernetes

Datasource(s)?

prometheus

[cazeaux]

Just an update to report that the issue is also present in Grafana 11.0.0

[cazeaux]

Hello

I have found where the regression happened. It has been introduced in v10.1.6, with this commit: https://github.com/grafana/grafana/commit/ae581286bb4894df7e75f1a48ec8a2a7f5f201be#diff-79a8e0c726aee282de46b93b30ea4022261351ea9c0d46e6f988296cfcf5c900

We see in this commit that the replace happens before the enhanceExprWithAdHocFilters. So this is why the variables are not replaces anymore.

The same inversion is also there in the last version, even if the code has significantly changed : https://github.com/grafana/grafana/blob/c73bbf19a1ed4f2f092a40922dda580775e424e4/packages/grafana-prometheus/src/datasource.ts#L844-L851

[tonypowa]

thanks for reporting this issue and for your contribution @cazeaux 🎉 I've added the issue to the team's board