InfluxDB: Flux unauthorized access

tonypowa commented 4 weeks ago

What happened?

I have a dashboard with two mainly 2 data sources: MySQL and InfluxDB.

Both worked fine when setting up the dashboard.

Now, InfluxDB graphs are not always working (9 times out of 10 not working), showing "Status: 500. Message: unauthorized: unauthorized access".

I have around 6 graphs with InfluxDB data source. Usually all of them are showing the above error. Sometimes only some of them are showing the error, others displayed fine.

Sometimes, if I refresh the page, some graphs will appear, some will disappear. It is really unpredictable.

If I check the data source on the Connections -> Data sources page and hit Save & Test, it is always working, no errors there.

What did you expect to happen?

Graphs should appear consistently, as before.

Did this work before?

Yes, it worked on previous versions (unfortunately, I don't know when it started to behave like this).

How do we reproduce it?

I'm unclear of the steps to reproduce it.

The dashboard code is provided below.

InfluxDB is really basic, no custom configuration.

Is the bug inside a dashboard panel?

| Key | Value |

|--|--|

| Panel | timeseries @ Grafana v11.1.0 (5b85c4c2fc) |

| Grafana | Grafana v11.1.0 (5b85c4c2fc) // Open Source |

Panel debug snapshot dashboard

```json { "panels": [ { "datasource": { "type": "grafana", "uid": "grafana" }, "description": "A hőmérséklet és páratartalom szenzor által visszajelzett érték", "fieldConfig": { "defaults": { "custom": { "drawStyle": "line", "lineInterpolation": "smooth", "barAlignment": 0, "lineWidth": 1, "fillOpacity": 0, "gradientMode": "none", "spanNulls": 1200000, "insertNulls": 960000, "showPoints": "auto", "pointSize": 5, "stacking": { "mode": "none", "group": "A" }, "axisPlacement": "auto", "axisLabel": "", "axisColorMode": "text", "axisBorderShow": false, "scaleDistribution": { "type": "linear" }, "axisCenteredZero": false, "hideFrom": { "tooltip": false, "viz": false, "legend": false }, "thresholdsStyle": { "mode": "off" }, "lineStyle": { "fill": "solid" } }, "color": { "mode": "palette-classic" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "celsius" }, "overrides": [ { "matcher": { "id": "byName", "options": "Humidity" }, "properties": [ { "id": "unit", "value": "humidity" } ] } ] }, "gridPos": { "h": 13, "w": 15, "x": 0, "y": 0 }, "id": 2, "options": { "tooltip": { "mode": "single", "sort": "none" }, "legend": { "showLegend": true, "displayMode": "list", "placement": "bottom", "calcs": [] } }, "targets": [ { "refId": "A", "datasource": { "type": "grafana", "uid": "grafana" }, "queryType": "snapshot", "snapshot": [ { "schema": { "refId": "A", "meta": { "typeVersion": [ 0, 0 ], "executedQueryString": "from(bucket: \"log\")\r\n |> range(start: 2024-06-10T23:02:07.703Z, stop: 2024-07-10T23:02:07.703Z)\r\n |> filter(fn: (r) => r[\"_measurement\"] == \"environment\" and r[\"key\"] == \"HU/BLAKE/U2-PT3.5\" and r[\"_field\"] == \"boardTemp\")\r\n |> aggregateWindow(every: 1s, fn: mean, createEmpty: false)\r\n |> map(fn: (r) => ({ time : r[\"_time\"], \"Board Temperature\" : r._value}))\r\n" }, "fields": [] }, "data": { "values": [] } }, { "schema": { "refId": "B", "meta": { "typeVersion": [ 0, 0 ], "executedQueryString": "from(bucket: \"log\")\r\n |> range(start: 2024-06-10T23:02:07.703Z, stop: 2024-07-10T23:02:07.703Z)\r\n |> filter(fn: (r) => r[\"_measurement\"] == \"environment\" and r[\"key\"] == \"HU/BLAKE/U2-PT3.5\" and r[\"_field\"] == \"coreTemp\")\r\n |> aggregateWindow(every: 1s, fn: mean, createEmpty: false)\r\n |> map(fn: (r) => ({ time : r[\"_time\"], \"Core Temperature\" : r._value}))\r\n" }, "fields": [] }, "data": { "values": [] } }, { "schema": { "refId": "C", "meta": { "typeVersion": [ 0, 0 ], "executedQueryString": "from(bucket: \"log\")\r\n |> range(start: 2024-06-10T23:02:07.703Z, stop: 2024-07-10T23:02:07.703Z)\r\n |> filter(fn: (r) => r[\"_measurement\"] == \"environment\" and r[\"key\"] == \"HU/BLAKE/U2-PT3.5\" and r[\"_field\"] == \"sensorTemp\")\r\n |> aggregateWindow(every: 1s, fn: mean, createEmpty: false)\r\n |> map(fn: (r) => ({ time : r[\"_time\"], \"Sensor Temperature\" : r._value}))" }, "fields": [] }, "data": { "values": [] } }, { "schema": { "refId": "D", "meta": { "typeVersion": [ 0, 0 ], "executedQueryString": "from(bucket: \"log\")\r\n |> range(start: 2024-06-10T23:02:07.703Z, stop: 2024-07-10T23:02:07.703Z)\r\n |> filter(fn: (r) => r[\"_measurement\"] == \"environment\" and r[\"key\"] == \"HU/BLAKE/U2-PT3.5\" and r[\"_field\"] == \"humidity\")\r\n |> aggregateWindow(every: 1s, fn: mean, createEmpty: false)\r\n |> map(fn: (r) => ({ time : r[\"_time\"], \"Humidity\" : r._value}))" }, "fields": [] }, "data": { "values": [] } } ] } ], "title": "Reproduced with embedded data", "type": "timeseries" }, { "gridPos": { "h": 7, "w": 9, "x": 15, "y": 0 }, "id": 5, "options": { "content": "\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

Panel	timeseries @ Grafana v11.1.0 (5b85c4c2fc)
Queries	A[influxdb], B[influxdb], C[influxdb], D[influxdb]
Data	Error 4 frames, 0 fields, 0 rows
Grafana	Grafana v11.1.0 (5b85c4c2fc) // Open Source

", "mode": "html" }, "title": "Debug info", "type": "text" }, { "id": 6, "title": "Original Panel JSON", "type": "text", "gridPos": { "h": 13, "w": 9, "x": 15, "y": 7 }, "options": { "content": "{\n \"datasource\": {\n \"type\": \"influxdb\",\n \"uid\": \"f67e8079-d4bc-426c-95d5-024e6b432506\"\n },\n \"description\": \"A hőmérséklet és páratartalom szenzor által visszajelzett érték\",\n \"fieldConfig\": {\n \"defaults\": {\n \"custom\": {\n \"drawStyle\": \"line\",\n \"lineInterpolation\": \"smooth\",\n \"barAlignment\": 0,\n \"lineWidth\": 1,\n \"fillOpacity\": 0,\n \"gradientMode\": \"none\",\n \"spanNulls\": 1200000,\n \"insertNulls\": 960000,\n \"showPoints\": \"auto\",\n \"pointSize\": 5,\n \"stacking\": {\n \"mode\": \"none\",\n \"group\": \"A\"\n },\n \"axisPlacement\": \"auto\",\n \"axisLabel\": \"\",\n \"axisColorMode\": \"text\",\n \"axisBorderShow\": false,\n \"scaleDistribution\": {\n \"type\": \"linear\"\n },\n \"axisCenteredZero\": false,\n \"hideFrom\": {\n \"tooltip\": false,\n \"viz\": false,\n \"legend\": false\n },\n \"thresholdsStyle\": {\n \"mode\": \"off\"\n },\n \"lineStyle\": {\n \"fill\": \"solid\"\n }\n },\n \"color\": {\n \"mode\": \"palette-classic\"\n },\n \"mappings\": [],\n \"thresholds\": {\n \"mode\": \"absolute\",\n \"steps\": [\n {\n \"color\": \"green\",\n \"value\": null\n },\n {\n \"color\": \"red\",\n \"value\": 80\n }\n ]\n },\n \"unit\": \"celsius\"\n },\n \"overrides\": [\n {\n \"matcher\": {\n \"id\": \"byName\",\n \"options\": \"Humidity\"\n },\n \"properties\": [\n {\n \"id\": \"unit\",\n \"value\": \"humidity\"\n }\n ]\n }\n ]\n },\n \"gridPos\": {\n \"h\": 8,\n \"w\": 8,\n \"x\": 0,\n \"y\": 18\n },\n \"id\": 57,\n \"options\": {\n \"tooltip\": {\n \"mode\": \"single\",\n \"sort\": \"none\"\n },\n \"legend\": {\n \"showLegend\": true,\n \"displayMode\": \"list\",\n \"placement\": \"bottom\",\n \"calcs\": []\n }\n },\n \"targets\": [\n {\n \"datasource\": {\n \"type\": \"influxdb\",\n \"uid\": \"f67e8079-d4bc-426c-95d5-024e6b432506\"\n },\n \"query\": \"from(bucket: \\\"log\\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r[\\\"_measurement\\\"] == \\\"environment\\\" and r[\\\"key\\\"] == \\\"${pedestal:text}\\\" and r[\\\"_field\\\"] == \\\"boardTemp\\\")\\r\\n |> aggregateWindow(every: ${interval}, fn: mean, createEmpty: false)\\r\\n |> map(fn: (r) => ({ time : r[\\\"_time\\\"], \\\"Board Temperature\\\" : r._value}))\\r\\n\",\n \"refId\": \"A\"\n },\n {\n \"datasource\": {\n \"type\": \"influxdb\",\n \"uid\": \"f67e8079-d4bc-426c-95d5-024e6b432506\"\n },\n \"hide\": false,\n \"query\": \"from(bucket: \\\"log\\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r[\\\"_measurement\\\"] == \\\"environment\\\" and r[\\\"key\\\"] == \\\"${pedestal:text}\\\" and r[\\\"_field\\\"] == \\\"coreTemp\\\")\\r\\n |> aggregateWindow(every: ${interval}, fn: mean, createEmpty: false)\\r\\n |> map(fn: (r) => ({ time : r[\\\"_time\\\"], \\\"Core Temperature\\\" : r._value}))\\r\\n\",\n \"refId\": \"B\"\n },\n {\n \"datasource\": {\n \"type\": \"influxdb\",\n \"uid\": \"f67e8079-d4bc-426c-95d5-024e6b432506\"\n },\n \"hide\": false,\n \"query\": \"from(bucket: \\\"log\\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r[\\\"_measurement\\\"] == \\\"environment\\\" and r[\\\"key\\\"] == \\\"${pedestal:text}\\\" and r[\\\"_field\\\"] == \\\"sensorTemp\\\")\\r\\n |> aggregateWindow(every: ${interval}, fn: mean, createEmpty: false)\\r\\n |> map(fn: (r) => ({ time : r[\\\"_time\\\"], \\\"Sensor Temperature\\\" : r._value}))\",\n \"refId\": \"C\"\n },\n {\n \"datasource\": {\n \"type\": \"influxdb\",\n \"uid\": \"f67e8079-d4bc-426c-95d5-024e6b432506\"\n },\n \"hide\": false,\n \"query\": \"from(bucket: \\\"log\\\")\\r\\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\\r\\n |> filter(fn: (r) => r[\\\"_measurement\\\"] == \\\"environment\\\" and r[\\\"key\\\"] == \\\"${pedestal:text}\\\" and r[\\\"_field\\\"] == \\\"humidity\\\")\\r\\n |> aggregateWindow(every: ${interval}, fn: mean, createEmpty: false)\\r\\n |> map(fn: (r) => ({ time : r[\\\"_time\\\"], \\\"Humidity\\\" : r._value}))\",\n \"refId\": \"D\"\n }\n ],\n \"title\": \"Hőmérséklet és páratartalom\",\n \"type\": \"timeseries\"\n}", "mode": "code", "code": { "language": "json", "showLineNumbers": true, "showMiniMap": true } } }, { "id": 3, "title": "Data from panel above", "type": "table", "datasource": { "type": "datasource", "uid": "-- Dashboard --" }, "gridPos": { "h": 7, "w": 15, "x": 0, "y": 13 }, "options": { "showTypeIcons": true }, "targets": [ { "datasource": { "type": "datasource", "uid": "-- Dashboard --" }, "panelId": 2, "withTransforms": true, "refId": "A" } ] } ], "schemaVersion": 37, "title": "Debug: Hőmérséklet és páratartalom // 2024-07-11 01:05:59", "tags": [ "debug", "debug-timeseries" ], "time": { "from": "2024-06-10T23:05:59.865Z", "to": "2024-07-10T23:05:59.865Z" } } ```

Environment (with versions)?

Grafana: v11.1.0 (5b85c4c2fc)

OS: Windows 11 (also checked: iOS, macOS)

Browser: Chrome (also checked: Safari, Firefox)

Grafana platform?

Docker

Datasource(s)?

influxdb

tonypowa commented 4 weeks ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#90313

moxious commented 4 weeks ago

Summary: InfluxDB graphs in a dashboard intermittently display an 'unauthorized access' error, despite data source tests passing. Issue remains open and unpredictable behavior occurs upon dashboard refreshes.

moxious commented 4 weeks ago

Elaboration:

Hi @tonypowa, thank you for taking the time to report this issue. To help us address this unpredictable behavior with your InfluxDB graphs, could you please provide a little more information? This will enable us to better understand and troubleshoot the problem you're encountering. Here are some questions and requests that could help us move forward:

InfluxDB Version: Can you specify the version of InfluxDB you're using?
Authentication Details: Without sharing any sensitive information, could you describe how the authentication for InfluxDB has been configured? Is there a possibility that tokens or credentials have been rotated or expired?
Grafana Logs: Could you provide relevant excerpts from the Grafana server logs when you encounter the unauthorized access error? It might offer additional clues about what's failing.
Network Requests: Using the developer tools in your browser, could you monitor and share the details of the network requests that result in a 500 error? Look especially for any request headers or response bodies that mention the unauthorized access.
Dashboard Configuration: Is there any custom configuration in your dashboard related to permissions or data source access that might affect the InfluxDB queries?
Steps to Reproduce: While you stated it's unclear how to reproduce the issue, any patterns or correlations you have noticed with the problem occurring (such as time of day, specific actions performed before the error, etc.) could be helpful.
InfluxDB Logs: If possible, checking the logs for the InfluxDB instance at times when the errors occur might also provide insight. Please share any relevant log messages.

Once we have these additional details, we'll be in a better position to understand what might be causing this sporadic unauthorized access error with InfluxDB on your Grafana dashboard.

Thanks again for your contribution to the community!

moxious commented 4 weeks ago

Hi @tonypowa, it seems like you are experiencing intermittent authorization issues with InfluxDB within Grafana. This could be related to the Grafana backend or datasource configuration. I would suggest transferring this issue to the AWS Datasources project as InfluxDB is one of the supported datasource plugins, and the team handling AWS Datasources should also be looking into issues related to other datasource plugins like InfluxDB.

Please ensure that your InfluxDB credentials have not changed and have the necessary permissions for the queries Grafana is attempting to execute. In the meantime, it might be helpful to check the Grafana server logs for any additional errors and to confirm if this behavior is consistent across different browsers or isolated to a specific environment.

moxious / triage