Grafana Alert Data

[FloHofstetter]

Environment

• Grafana version: 10.4.1
• Databricks plugin version: 1.2.4
• Browser: Google Chrome
• DBSQL version: 2023.50

Issue Description

When executing queries through the normal Grafana dashboard, the Databricks data source returns expected results and operates without issues. However, utilizing the identical query within the Grafana Alert Manager results in a response lacking meaningful data.

Steps to Reproduce

1. Configure a data source in Grafana to connect to Databricks.
2. Create a dashboard with a query to Databricks that successfully retrieves and displays data.
3. Use the same query in the Grafana Alert Manager.
4. Observe the lack of meaningful data in the alert manager's output, despite the query working in the dashboard.

Expected Behavior

The Grafana Alert Manager should return meaningful data similar to the dashboard when using the same queries.

Actual Behavior

No meaningful data is returned in the Alert Manager, despite the query executing successfully in the Databricks query history for both dashboard and alert contexts.

Additional Information

• The Databricks query history shows the query executes successfully in both dashboard and alert contexts with no apparent errors.

Possible Impact

This issue prevents the effective use of Grafana alerts with Databricks as a data source, potentially impacting monitoring and alerting for important metrics.

[mullerpeter]

Hi @FloHofstetter thanks for opening the issue. I'm not really able to reproduce the issue. For my test query the alerting query works fine and returns the same data as in the dashboard.

Could you try the same query as me using the samples dataset (which should be available on every Databricks instance) to check if the issue still persists.

My Time Range: From now-924021753s To now-900520953s

SELECT window.start, float(avg(o_totalprice))
FROM samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY $__timeWindow(o_orderdate)
ORDER BY start ASC;

Edit: Or this one which is almost identical to your query and also works for me:

SELECT 
  timestamp(o_orderdate), 
  float(avg(o_totalprice))
FROM 
  samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY o_orderdate;

[FloHofstetter]

Hi @mullerpeter,

thank you for your quick response. Here are my results:

SELECT window.start, float(avg(o_totalprice))
FROM samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY $__timeWindow(o_orderdate)
ORDER BY start ASC;

SELECT 
  timestamp(o_orderdate), 
  float(avg(o_totalprice))
FROM 
  samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY o_orderdate;

I used the time range: From now-924021753s To now-900520953s

So the first query has the same result as you describe. Thes second one has the same behaviour as I discovered with my real data.

[mullerpeter]

Ok so I was able to reproduce the issue you are having. Seems as if Grafana has an issue with the unsorted data in the Alerting. If I add an ORDER BY to the end of the query it works.

SELECT 
  timestamp(o_orderdate), 
  float(avg(o_totalprice))
FROM 
  samples.tpch.orders
WHERE $__timeFilter(o_orderdate)
GROUP BY o_orderdate
ORDER BY o_orderdate;

Could you try your original query with an ORDER BY measured_at at the end to see if that works for you too.

I'll need to dig a bit deeper into why this is an issue, but in the meantime this could fix your problem.

[FloHofstetter]

Hi @mullerpeter,

Thank you for your response. Changing the order resolved the issue, and I can now see the data in the graph within the alerting view.

I removed the float cast due to the presence of Null values in the columns. From my perspective, this issue is resolved. Please feel free to close it, or keep it open if you wish to explore further the ordering requirements of Grafana.

Best regards, Florian