Alerting with OpenTSDB, graph data not sorted by time

[NikolaBorisov]

What happened: Data from query on the Create Alert Rule page is not displayed properly. It looks like the data is not sorted and the graph looks terrible.

What you expected to happen: I expected the graph displayed on the Create alert rule page to look exactly like the panel for which I'm trying to create the alert.

How to reproduce it (as minimally and precisely as possible): Try to create an alert on a OpenTSDB graph panel. Click "Run queries" and look at the data in Time series format and also table format.

Anything else we need to know?:

Environment:

• Grafana version: 8.1.2
• Data source type & version: OpenTSDB 2.4
• OS Grafana is installed on: Ubuntu
• User OS & Browser: Ubuntu / Firefox
• Grafana plugins:
• Others:

[wbrowne]

I believe I'm seeing something similar

[kylebrandt]

It looks like on the backend for OpenTSDB, response object from OpenTSDB is a map. Since server side expressions go through the backend, we do not get sorted data (since old alerting was always reducing, order didn't really matter).

IIRC, opentsdb has an option that is (asArray) or something like this, so the request should specify that, and then we would get ordered data back.

Edit: I think "arrays" under "api/query": http://opentsdb.net/docs/build/html/api_http/serializers/json.html

[peterholmberg]

@kylebrandt this sounds like a change to the opentsdb plugin then? Would it change anything for paneledit/explore if we enabled arrays?

[kylebrandt]

@peterholmberg No, this change would entirely be on the backend side and within the OpenTSDB datasource.

[wbrowne]

Then this is most likely related to changes here I imagine https://github.com/grafana/grafana/pull/35815

[kylebrandt]

@wbrowne The following needs to change to not be a map for Datapoints:

https://github.com/grafana/grafana/blob/7e6cf14c84fd3bbd1ba709ecb772ff7e3afd4d6a/pkg/tsdb/opentsdb/types.go#L9-L12

And then however you specify that serializer option in my link above would also need to be set so the response marshals into it.

[kylebrandt]

@wbrowne Also if you look at http://opentsdb.net/docs/build/html/api_http/query/index.html, the response contains tags, so those should be captured in the Labels property of the numeric field so things work correctly with alerting.

e.g. the host tag in the following:

[
    {
        "metric": "tsd.hbase.puts",
        "tags": {
            "host": "tsdb-1.mysite.com"
        },
        "aggregatedTags": [],
        "dps": {
            "1365966001": 3758788892,
            "1365966061": 3758804070,
...
            "1365974281": 3778141673
        }
    },
    {
        "metric": "tsd.hbase.puts",
        "tags": {
            "host": "tsdb-2.mysite.com"
        },
        "aggregatedTags": [],
        "dps": {
            "1365966001": 3902179270,
            "1365966062": 3902197769,
...
            "1365974281": 3922266478
        }
    }
]

[NikolaBorisov]

Hello All,

Is there any chance to fix this? Basically the new alerting does not work for people using OpenTSDB...

[wbrowne]

CC @grafana/observability-squad just in case this one is lost

[NikolaBorisov]

Any change someone can look at this one.

Because of this we can not use grafana alerts with OpenTSDB

[ivanahuckova]

+cc @grafana/observability-metrics

[joey-grafana]

Hey OpenTSDB is owned by the metrics squad, not the traces and profiling squad :) I'm unassigning myself and you already have someone from the metrics squad!

[aangelisc]

This should be fixed by #90991