search

Altinity / clickhouse-grafana

Altinity Grafana datasource plugin for ClickHouse®
MIT License
720 stars 120 forks source link

$columns + ORDER BY ... WITH FILL ... doesn't work in alerts #409

Closed Enton-72 closed 6 months ago

Enton-72 commented 2 years ago

Hello everyone,

I have an issue while trying to create alert rule on Grafana using this plugin. Issue is that different graphs are plotted using same query on "Create alert rule" and "Explore" sections. Because of this i am unable to create multi dimensional rule as described here.

In Explore section graph is plotted correctly and requests are grouped by status code as thay should.

Screenshot 2022-02-23 at 10 58 31

But, using same query in Create alert rule section i got strange grouping which i don't understand.

Screenshot 2022-02-23 at 11 00 17

I have tried the same thing with InfluxDB as a source and graphs were indentical.

I am not sure why it is behaving like this so any insight is welcome.

Query:

SELECT 
    $timeSeries as t,
    sum(request) as request,
    status_code
FROM $table
WHERE
    $timeFilter
    AND status_code != 201 AND status_code != 0
GROUP BY t, status_code
ORDER BY t WITH FILL STEP 60000

ClickHouse: 21.11.6.7 (official build) Grafana version: v8.4.1 (53f5c6a44c) Plugin version: v2.4.2 Plugin was installed using grafana-cli command. ClickHouse and Grafana are installed on Debian 11

Slach commented 2 years ago

Could you try to use following query

$columns(
 status_code,
 sum(request) as sum_req
)
FROM $table  
WHERE
    $timeFilter
    AND status_code != 201 AND status_code != 0
GROUP BY t, status_code
ORDER BY t WITH FILL STEP 60000

Look details on https://github.com/Vertamedia/clickhouse-grafana/tree/master/#columnskey-value---query-values-as-array-of-key-value-where-key-will-be-used-as-label

Enton-72 commented 2 years ago

Thanks for quick reply.

I am getting an error:

Failed to evaluate queries and expressions: failed to execute conditions: failed to execute query A: 
Code: 62. DB::Exception: Syntax error: failed at position 615 ('GROUP') (line 6, col 33): 
GROUP BY t, status_code ORDER BY t, status_code) GROUP BY t ORDER BY t FORMAT JSON /* alerts query */; . 
Expected one of: UNION, LIMIT, DoubleColon, LIKE, GLOBAL NOT IN, AS, DIV, IS, OR, EXCEPT, QuestionMark, OFFSET,
BETWEEN, NOT LIKE, MOD, AND, Comma, alias, SETTINGS, IN, ILIKE, INTERSECT, NOT ILIKE, NOT, token, NOT IN,
GLOBAL IN. (SYNTAX_ERROR) (version 21.11.6.7 (official build))
Slach commented 2 years ago

hm, could you click to "Generated SQL" and share generated query?

Enton-72 commented 2 years ago

When i click on "Generated SQL" it shows me only "true".

Screenshot 2022-02-23 at 12 25 02

Is there another way to see generated query maybe?

Slach commented 2 years ago

Could you check Generated SQL it on Graph visualization not on alerts?

Enton-72 commented 2 years ago

Yes. Output:

SELECT t,
       groupArray((status_code, sum_req)) AS groupArr
FROM
  (SELECT (intDiv(toUInt32(event_time), 2) * 2) * 1000 AS t,
          status_code,
          sum(request) AS sum_req
   FROM db.table
   WHERE event_date >= toDate(1645612422)
     AND event_date <= toDate(1645616022)
     AND event_time >= toDateTime(1645612422)
     AND event_time <= toDateTime(1645616022)
     AND event_date >= toDate(1645612422)
     AND event_date <= toDate(1645616022)
     AND event_time >= toDateTime(1645612422)
     AND event_time <= toDateTime(1645616022)
     AND status_code != 201
     AND status_code != 0
   GROUP BY t,
            status_code
   ORDER BY t WITH FILL STEP 60000
   GROUP BY t,
            status_code
   ORDER BY t,
            status_code)
GROUP BY t
ORDER BY t
Slach commented 2 years ago

ok. thanks a lot for reporting, I will try to fix $columns + WITH FILL STEP macros usage

you could try to use the following SQL query

SELECT t,
       groupArray((status_code, sum_req)) AS groupArr
FROM
(
   SELECT $timeSeries AS t,
          status_code,
          sum(request) AS sum_req
   FROM $table
   WHERE $timeFilter
     AND status_code != 201
     AND status_code != 0
   GROUP BY t,status_code
   ORDER BY t WITH FILL STEP 60000
)
GROUP BY t
ORDER BY t
Enton-72 commented 2 years ago

Glad i could help.

I tried the query and it works on Explorer, but on Create alert rule section doesn't work.

Explorer section:

Screenshot 2022-02-23 at 13 02 59

Create alert rule section:

Screenshot 2022-02-23 at 13 02 32
Slach commented 2 years ago

could you try setup GF_LOG_LEVEL=debug or change grafana.ini log section https://grafana.com/docs/grafana/latest/administration/configuration/#log

and try to share logs from systemctl restart grafana-server && journalctl -xue --no-pager -u grafana-server

Enton-72 commented 2 years ago

Command:

export GF_LOG_LEVEL=debug && systemctl restart grafana-server && journalctl -xue --no-pager -u grafana-server

Output:

-- Journal begins at Wed 2022-02-23 13:19:32 UTC, ends at Wed 2022-02-23 13:19:32 UTC. --
Feb 23 13:19:32 dev-server systemd[1]: Stopping Grafana instance...
░░ Subject: A stop job for unit grafana-server.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit grafana-server.service has begun execution.
░░ 
░░ The job identifier is 1256.
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=info msg="Shutdown started" reason="System signal: terminated"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*updatechecker.PluginsService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*metrics.InternalMetricsService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*notifications.NotificationService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*pushhttp.Gateway reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*remotecache.RemoteCache reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*updatechecker.GrafanaService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=secrets t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="grafana is shutting down; stopping..."
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*live.GrafanaLive reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*manager.SecretsService reason=null
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*service.UsageStats reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*rendering.RenderingService reason=null
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=tracing t=2022-02-23T13:19:32.23+0000 lvl=info msg="Closing tracing"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*auth.UserAuthTokenService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.postgres t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=provisioning t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stop polling for dashboard changes"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*cleanup.CleanUpService reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertGroups t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=ngalert t=2022-02-23T13:19:32.23+0000 lvl=eror msg="unable to fetch orgIds" msg="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.influxdb t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.gauge t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.table t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.grafana t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.loki t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.stackdriver t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.tempo t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.opentsdb t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.stat t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.pluginlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.welcome t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.timeseries t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.barchart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.debug t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.prometheus t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mysql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.histogram t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.barchart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.annolist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.state-timeline t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mysql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.annolist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.live t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.status-history t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.cloudwatch t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.logs t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.state-timeline t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.grafana-azure-monitor-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.stackdriver t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.logs t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertGroups t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.zipkin t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.debug t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.zipkin t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.postgres t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.influxdb t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*tracing.Opentelemetry reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.geomap t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertmanager t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="plugin process exited" path=/var/lib/grafana/plugins/vertamedia-clickhouse-datasource/vertamedia-clickhouse-plugin_linux_amd64 pid=13914 error="signal: terminated"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.alertmanager t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.testdata t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.testdata t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=http.server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="server was shutdown gracefully"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.elasticsearch t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.tempo t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.xychart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.stat t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.xychart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mssql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.dashlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.canvas t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.timeseries t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.canvas t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.histogram t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.table-old t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.graphite t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.text t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.grafana t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.prometheus t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.graph t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.graphite t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.nodeGraph t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.graph t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=infra.kvstore.sql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="kvstore value not changed" orgId=1 namespace=alertmanager key=notifications value=Rwo3CgV7fTp7fRIUCgtFbnRvbi1TbGFjaxIFc2xhY2sqDAjBs9iQBhDT8quxAToK1JWHzbvbwdODARIMCMHi8pAGENPyq7EB
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.table-old t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.opentsdb t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.gauge t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.text t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.jaeger t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=ngalert t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="stopping state cache metrics recording" now=2022-02-23T13:19:32.238069437Z
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.news t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.table t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.status-history t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.pluginlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.gettingstarted t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.gettingstarted t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="plugin exited"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.live t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.input t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.input t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.dashboard t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.bargauge t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*provisioning.ProvisioningServiceImpl reason="context canceled"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.grafana-azure-monitor-datasource t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.nodeGraph t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.icon t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.loki t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.icon t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.elasticsearch t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.heatmap t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.piechart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.candlestick t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=infra.kvstore.sql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="kvstore value not changed" orgId=1 namespace=alertmanager key=silences value=
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.heatmap t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mixed t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopping plugin"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.geomap t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*api.HTTPServer reason=null
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mssql t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.dashlist t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.welcome t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.jaeger t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.news t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.dashboard t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.bargauge t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.cloudwatch t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.piechart t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.candlestick t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*ngalert.AlertNG reason=null
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=plugin.mixed t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Plugin stopped"
Feb 23 13:19:32 dev-server grafana-server[13893]: logger=server t=2022-02-23T13:19:32.23+0000 lvl=dbug msg="Stopped background service" service=*manager.PluginManager reason="context canceled"
Feb 23 13:19:32 dev-server systemd[1]: grafana-server.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit grafana-server.service has successfully entered the 'dead' state.
Feb 23 13:19:32 dev-server systemd[1]: Stopped Grafana instance.
░░ Subject: A stop job for unit grafana-server.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit grafana-server.service has finished.
░░ 
░░ The job identifier is 1256 and the job result is done.
Feb 23 13:19:32 dev-server systemd[1]: grafana-server.service: Consumed 1.840s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit grafana-server.service completed and consumed the indicated resources.
Feb 23 13:19:32 dev-server systemd[1]: Started Grafana instance.
░░ Subject: A start job for unit grafana-server.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit grafana-server.service has finished successfully.
░░ 
░░ The job identifier is 1256.
Slach commented 2 years ago

hmm, look like only stop was logged

try run journalctl -xue --no-pager -u grafana-server again?

Enton-72 commented 2 years ago

server.log

Slach commented 2 years ago

@Enton-72 looks strange, i don't see any error related to SQL from vertamedia backend part of plugin plugin succesfully loaded

Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.16+0000 lvl=dbug msg="starting plugin" path=/var/lib/grafana/plugins/vertamedia-clickhouse-datasource/vertamedia-clickhouse-plugin_linux_amd64 args="unsupported value type"
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.16+0000 lvl=dbug msg="plugin started" path=/var/lib/grafana/plugins/vertamedia-clickhouse-datasource/vertamedia-clickhouse-plugin_linux_amd64 pid=13976
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.16+0000 lvl=dbug msg="waiting for RPC address" path=/var/lib/grafana/plugins/vertamedia-clickhouse-datasource/vertamedia-clickhouse-plugin_linux_amd64
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.17+0000 lvl=info msg="Starting ClickHouse datasource backend..."
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.17+0000 lvl=dbug msg="Serving plugin" plugins="unsupported value type"
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.17+0000 lvl=dbug msg="using plugin" version=2
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.17+0000 lvl=dbug msg="plugin address" address=/tmp/plugin2059328789 network=unix
Feb 23 13:19:33 dev-server grafana-server[13955]: logger=plugin.vertamedia-clickhouse-datasource t=2022-02-23T13:19:33.18+0000 lvl=dbug msg="Successfully started backend plugin process"

After it ngalert found zero alerts rules

Feb 23 13:19:43 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:19:43.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"
Feb 23 13:19:48 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:19:48.39+0000 lvl=dbug msg="recording state cache metrics" now=2022-02-23T13:19:48.397965973Z
Feb 23 13:19:53 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:19:53.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"
Feb 23 13:20:03 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:03.39+0000 lvl=dbug msg="recording state cache metrics" now=2022-02-23T13:20:03.397900578Z
Feb 23 13:20:03 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:03.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"
Feb 23 13:20:13 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:13.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"
Feb 23 13:20:18 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:18.39+0000 lvl=dbug msg="recording state cache metrics" now=2022-02-23T13:20:18.397999239Z
Feb 23 13:20:23 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:23.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"
Feb 23 13:20:33 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:33.39+0000 lvl=dbug msg="recording state cache metrics" now=2022-02-23T13:20:33.398122009Z
Feb 23 13:20:33 dev-server grafana-server[13955]: logger=ngalert t=2022-02-23T13:20:33.39+0000 lvl=dbug msg="alert rules fetched" count=0 disabled_orgs="unsupported value type"

only one stange thing disabled_orgs="unsupported value type" but i'm not sure it related to your shared screentshot

could you try add query to alerts again, see error message in browser and look to journalctl -xue --no-pager -u grafana-server | grep -i -E "verta|click|alert" again?

Enton-72 commented 2 years ago

I have added 2 alerts, restarted server and tried to execute queries in Alert section.

Screenshot 2022-02-23 at 16 12 33

After that i executed command:

journalctl -xue --no-pager -u grafana-server | grep -i -E "verta|click|alert"

And output is: server.log

Slach commented 2 years ago

ok. got it

panic: interface conversion: main.Value is float64, not string
goroutine 88 [running]:
main.(*Response).toFramesWithTimeStamp(0xc000480740, 0x9f97b2, 0xc0001626c0, 0x0, 0xc000465f80, 0x0)
        /home/runner/work/clickhouse-grafana/clickhouse-grafana/pkg/response.go:119 +0xd1b
main.(*Response).toFrames(0xc000010728, 0xc4d5b0, 0xc000480380)
        /home/runner/work/clickhouse-grafana/clickhouse-grafana/pkg/response.go:34 +0x65
main.(*ClickHouseDatasource).executeQuery(0xc00035e6e0, {0x1, {0xc00003b840, 0x20}, 0x0, 0x0, 0xc00013dc30}, {0xc4d5b0, 0xc000480380}, 0xc000675e28)
        /home/runner/work/clickhouse-grafana/clickhouse-grafana/pkg/datasource.go:57 +0x146
main.(*ClickHouseDatasource).evalQuery(0x17, {0x1, {0xc00003b840, 0x20}, 0x0, 0x0, 0xc00013dc30}, {0xc4d5b0, 0xc000480380}, 0xc000294800)
        /home/runner/work/clickhouse-grafana/clickhouse-grafana/pkg/datasource.go:84 +0x23b
main.(*ClickHouseDatasource).QueryData.func3()
        /home/runner/work/clickhouse-grafana/clickhouse-grafana/pkg/datasource.go:121 +0xae
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x67
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:54 +0x92

will try to figure out

Slach commented 2 years ago

@Enton-72 could you try to use following query? toString(status_code) should allow us handle query properly

SELECT t,
       groupArray((toString(status_code), sum_req)) AS groupArr
FROM
(
   SELECT $timeSeries AS t,
          status_code,
          sum(request) AS sum_req
   FROM $table
   WHERE $timeFilter
     AND status_code != 201
     AND status_code != 0
   GROUP BY t,status_code
   ORDER BY t WITH FILL STEP 60000
)
GROUP BY t
ORDER BY t
Enton-72 commented 2 years ago

Yess, it works now.

Screenshot 2022-02-23 at 17 04 23

One more thing if you can halp me please. On picture you can see status_code tags are duplicated, is there a way to remove duplicates?

Enton-72 commented 2 years ago

I got another issue while trying to create multi dimensional rule. Error: invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {}

Screenshot 2022-02-24 at 08 58 59 Screenshot 2022-02-24 at 08 59 17 Screenshot 2022-02-24 at 08 58 41

I sow that others had issue whit this using prometheus db (Issue). I tried their solution here but with no success. Also tried to add GROUP BY t, status_code at the end of query, as described here, but nothings changed. I am not sure what else i could try.

Debug message while executing "Preview alerts" button:

logger=datasources t=2022-02-24T08:12:10.49+0000 lvl=dbug msg="Querying for data source via SQL store" uid=wpp09bf7z orgId=1
logger=plugin.vertamedia-clickhouse-datasource t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="queryResponse: " SELECTt,groupArray((toString(status_code),sum_req))ASgroupArrFROM(SELECT(intDiv(toUInt32(event_time),60)*60)*1000ASt,status_code,sum(request)ASsum_reqFROMdb.tableWHEREevent_date>toDate(1645688530)ANDevent_date<toDate(1645690330)ANDevent_time>toDateTime(1645688530)ANDevent_time<toDateTime(1645690330)ANDstatus_code!201ANDstatus_code!0GROUPBYt,status_codeORDERBYtWITHFILLSTEP60000)GROUPBYtORDERBYtFORMATJSON/*alertsquery*/;="unsupported value type"
logger=expr t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="expression datasource query (seriesSet)" query=main
logger=expr t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="expression datasource query (seriesSet)" query=main
logger=expr t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="expression datasource query (seriesSet)" query=main
logger=expr t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="expression datasource query (seriesSet)" query=main
logger=expr t=2022-02-24T08:12:10.52+0000 lvl=dbug msg="expression datasource query (seriesSet)" query=main
Slach commented 2 years ago

could you try to another type of datasource? https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/ ?

Enton-72 commented 2 years ago

I tried but i got panic.

Query:

SELECT t,
       groupArray((toString(status_code), sum_req)) AS groupArr
FROM
(
   SELECT (intDiv(toUInt32(event_time), 2) * 2) * 1000 AS t,
          status_code,
          sum(request) AS sum_req
   FROM db.table
   WHERE $__timeFilter(event_time)
     AND status_code != 201
     AND status_code != 0
   GROUP BY t,status_code
   ORDER BY t WITH FILL STEP 60000
)
GROUP BY t
ORDER BY t

Debug log:

logger=datasources t=2022-02-24T10:00:56.05+0000 lvl=dbug msg="Querying for data source via SQL store" id=6 orgId=1
logger=query_data t=2022-02-24T10:00:56.05+0000 lvl=dbug msg="Processing metrics query" query="unsupported value type"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="panic: runtime error: invalid memory address or nil pointer dereference"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xa8b1f5]"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg=
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="goroutine 76 [running]:"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="github.com/grafana/sqlds.getFrames(0xc000356080, 0xd26948, {0xc0001e6000, 0xc0002f21c0, 0x1b6}, 0x0, 0xc00029e280)"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/sqlds@v1.3.0/query.go:128 +0x55"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="github.com/grafana/sqlds.query(0xd26de0, {0xc0001e6000, 0x24, 0x44}, 0x750, 0xc00029e280)"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/sqlds@v1.3.0/query.go:118 +0x72e"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="github.com/grafana/sqlds.(*sqldatasource).handleQuery(0xc0004b02a0, {{0x1211e48, 0x1}, {0xc0004b62c0, 0x3}, 0x750, 0x77359400, {{0x2ef7ed00, 0xed9a94047, 0x126c420}, ...}, ...})"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/sqlds@v1.3.0/datasource.go:105 +0x1e5"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="github.com/grafana/sqlds.(*sqldatasource).QueryData.func1({{0x1211e48, 0x1}, {0xc0004b62c0, 0x3}, 0x750, 0x77359400, {{0x2ef7ed00, 0xed9a94047, 0x126c420}, {0x2ef7ed00, ...}}, ...})"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/sqlds@v1.3.0/datasource.go:65 +0x78"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="created by github.com/grafana/sqlds.(*sqldatasource).QueryData"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="\t/root/go/pkg/mod/github.com/grafana/sqlds@v1.3.0/datasource.go:64 +0x2c5"
logger=context t=2022-02-24T10:00:56.12+0000 lvl=eror msg="Query data error" error="failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=192.168.1.2
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:56.12+0000 lvl=dbug msg="plugin process exited" path=/var/lib/grafana/plugins/grafana-clickhouse-datasource/gpx_clickhouse_linux_amd64 pid=31736 error="exit status 2"
logger=context t=2022-02-24T10:00:56.12+0000 lvl=eror msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.1.2 time_ms=73 size=30 referer="http://10.1.1.2/explore?orgId=1&left=%7B%22datasource%22:%22ClickHouseDirectNode%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22queryType%22:%22sql%22,%22rawSql%22:%22SELECT%20t,%5Cn%20%20%20%20%20%20%20groupArray((toString(status_code),%20sum_req))%20AS%20groupArr%5CnFROM%5Cn(%5Cn%20%20%20SELECT%20(intDiv(toUInt32(event_time),%202)%20*%202)%20*%201000%20AS%20t,%5Cn%20%20%20%20%20%20%20%20%20%20status_code,%5Cn%20%20%20%20%20%20%20%20%20%20sum(request)%20AS%20sum_req%5Cn%20%20%20FROM%20db.table%5Cn%20%20%20WHERE%20$__timeFilter(event_time)%5Cn%20%20%20%20%20AND%20status_code%20!%3D%20201%5Cn%20%20%20%20%20AND%20status_code%20!%3D%200%5Cn%20%20%20GROUP%20BY%20t,status_code%5Cn%20%20%20ORDER%20BY%20t%20WITH%20FILL%20STEP%2060000%5Cn)%5CnGROUP%20BY%20t%5CnORDER%20BY%20t%22,%22meta%22:%7B%22builderOptions%22:%7B%22mode%22:%22list%22,%22fields%22:%5B%5D,%22limit%22:100%7D%7D,%22format%22:1%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D"

logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.08+0000 lvl=dbug msg="Restarting plugin"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.08+0000 lvl=dbug msg="starting plugin" path=/var/lib/grafana/plugins/grafana-clickhouse-datasource/gpx_clickhouse_linux_amd64 args="unsupported value type"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.08+0000 lvl=dbug msg="plugin started" path=/var/lib/grafana/plugins/grafana-clickhouse-datasource/gpx_clickhouse_linux_amd64 pid=31749
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.08+0000 lvl=dbug msg="waiting for RPC address" path=/var/lib/grafana/plugins/grafana-clickhouse-datasource/gpx_clickhouse_linux_amd64
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.09+0000 lvl=info msg=Profiler enabled=false
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.09+0000 lvl=dbug msg="Serving plugin" plugins="unsupported value type"
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.09+0000 lvl=dbug msg="using plugin" version=2
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.09+0000 lvl=dbug msg="plugin address" network=unix address=/tmp/plugin2887451648
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T10:00:57.09+0000 lvl=dbug msg="Plugin restarted"

Also grafana-clickhouse-datasource plugin won't connect to clickhouse proxy, it only works when i connect grafana directly to clickhouse node.

Error while trying to connect to clickhouse proxy:

logger=datasources t=2022-02-24T09:36:22.35+0000 lvl=dbug msg="Received command to update data source" url=
logger=datasource t=2022-02-24T09:36:22.35+0000 lvl=dbug msg="Applying default URL parsing for this data source type" type=grafana-clickhouse-datasource url=
logger=datasource t=2022-02-24T09:36:22.35+0000 lvl=dbug msg="Data source URL doesn't specify protocol, so prepending it with http:// in order to make it unambiguous" type=grafana-clickhouse-datasource url=
logger=auth t=2022-02-24T09:36:22.36+0000 lvl=dbug msg="token needs rotation" tokenId=1 authTokenSeen=false rotatedAt=2022-02-24T09:35:19Z
logger=auth t=2022-02-24T09:36:22.37+0000 lvl=dbug msg="auth token rotated" affected=1 auth_token_id=1 userId=1
logger=auth t=2022-02-24T09:36:22.39+0000 lvl=dbug msg="seen token" tokenId=1 userId=1 clientIP=192.168.1.2 userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.109 Safari/537.36" authToken=b9ea484702c214abfe0b2d84e9d78e67749b7df73e32df96f4887acb2c202082
logger=plugin.grafana-clickhouse-datasource t=2022-02-24T09:36:22.75+0000 lvl=eror msg="[hello] unexpected packet [72] from server"

After changing a query a few times i ended up with query that can show data as time series.

SELECT $__timeInterval(event_time), status_code, sum(request) 
FROM ocamba.push_task_delivery 
WHERE ( event_date >= today() ) 
GROUP BY status_code, event_time 
ORDER BY event_time desc
Screenshot 2022-02-24 at 12 33 56

But still unusable for creating multi dimensional rules, because error is still present:

invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {}
kleemaj commented 2 years ago

I'm having the same issue with multidimensional rules. They do work correctly when using a classic condition, but not a math or reduce expression.

Slach commented 2 years ago

@kuri-man could you provide your query and Generated SQL?

kleemaj commented 2 years ago

@Slach I've written an example below.

Create a timeseries with some category that we want to group on:

DROP TABLE IF EXISTS test_data

CREATE TABLE IF NOT EXISTS test_data
(
    timestamp DateTime,
    category LowCardinality(String),
    value UInt64
) ENGINE = MergeTree ORDER BY timestamp

INSERT INTO test_data VALUES
    (now(), 'A', 1),
    (now(), 'B', 2),
    (now() - INTERVAL 10 MINUTE, 'A', 2),
    (now() - INTERVAL 15 MINUTE, 'B', 5),
    (now() - INTERVAL 60 MINUTE, 'A', 4)

Produce a query:

$columns(category, sum(value) AS c)
FROM test_data
WHERE $timeFilter(timestamp)

Generated SQL:

SELECT t, groupArray((category, c)) AS groupArr FROM ( SELECT (intDiv(toUInt32(timestamp), 15) * 15) * 1000 AS t, category, sum(value) AS c FROM test_data
WHERE timestamp >= toDateTime(1646641819) AND timestamp <= toDateTime(1646663419) AND timestamp >= toDateTime(1646641819) AND timestamp <= toDateTime(1646663419)(timestamp) GROUP BY t, category ORDER BY t, category) GROUP BY t ORDER BY t

I'd like to create an alert for if the total of value over the last 30 minutes is >5 for any one of the series. I did this with a reduce + math expression. image

The sum by series appears to work correctly, but I notice that they're each displayed by the query name rather than by the category. When I try to evaluate this, I get an error that @Enton-72 hit: image

When using a classic condition the alerting does work as expected. However, there are some limitations to that approach (any single label triggering the alert will prevent new alert updates when a second label would trigger it). I'm running plugin version 2.4.3, grafana 8.4.3.