search

divinity666 / ruby-grafana-reporter

Reporting Service for Grafana
MIT License
66 stars 5 forks source link

grafana_panel_query_table can not get data #24

Closed myselfghost closed 1 year ago

myselfghost commented 2 years ago

Environment (please complete the following information):

panel JSON：

{
  "id": 11,
  "gridPos": {
    "h": 18,
    "w": 24,
    "x": 0,
    "y": 0
  },
  "type": "table",
  "title": "Host Reporter",
  "transformations": [
    {
      "id": "seriesToColumns",
      "options": {
        "byField": "instance"
      }
    },
    {
      "id": "organize",
      "options": {
        "excludeByName": {
          "InstanceId": true,
          "InstanceId 2": true,
          "InstanceName": true,
          "InstanceName 2": true,
          "OsType": true,
          "OsType 2": true,
          "Time": true,
          "Time 2": true,
          "Time 3": true,
          "Time 4": true,
          "alertsource": true,
          "alertsource 2": true,
          "environment": true,
          "environment 2": true,
          "environment 3": true,
          "environment 4": true,
          "idc 1": true,
          "idc 2": true,
          "idc 3": true,
          "idc 4": true,
          "important": true,
          "important 2": true,
          "instanceid": true,
          "instanceid 2": true,
          "instancename 2": true,
          "instancename 3": true,
          "instancename 4": true,
          "job": true,
          "job 2": true,
          "monitor_group": true,
          "monitor_group 2": true,
          "no_mem_warn 1": true,
          "no_mem_warn 2": true,
          "ostype": true,
          "ostype 2": true,
          "system 2": true,
          "system 3": true,
          "system 4": true
        },
        "indexByName": {
          "InstanceId": 1,
          "InstanceName": 2,
          "OsType": 3,
          "Time": 0,
          "Value": 13,
          "alertsource": 4,
          "environment": 6,
          "idc": 7,
          "instance": 5,
          "instanceid": 8,
          "instancename": 9,
          "job": 10,
          "ostype": 11,
          "system": 12
        },
        "renameByName": {
          "Value": "mem_max",
          "Value #A": "mem_max(%)",
          "Value #B": "mem_avg(%)",
          "Value #C": "cpu_max(%)",
          "Value #D": "cpu_avg(%)",
          "instance": "IP"
        }
      }
    }
  ],
  "pluginVersion": "8.3.5",
  "fieldConfig": {
    "defaults": {
      "custom": {
        "align": "center",
        "displayMode": "auto"
      },
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "green",
            "value": null
          }
        ]
      },
      "mappings": [],
      "color": {
        "mode": "thresholds"
      },
      "max": 100,
      "min": 0
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "mem_avg(%)"
        },
        "properties": [
          {
            "id": "thresholds",
            "value": {
              "mode": "percentage",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "yellow",
                  "value": 80
                }
              ]
            }
          },
          {
            "id": "custom.displayMode",
            "value": "color-background"
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "cpu_avg(%)"
        },
        "properties": [
          {
            "id": "thresholds",
            "value": {
              "mode": "percentage",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          {
            "id": "custom.displayMode",
            "value": "color-background"
          }
        ]
      }
    ]
  },
  "options": {
    "showHeader": true,
    "footer": {
      "show": false,
      "reducer": [
        "sum"
      ],
      "fields": ""
    }
  },
  "targets": [
    {
      "expr": "100- 100* min_over_time(node_memory_MemAvailable_bytes{idc=\"$idc\",system=\"$system\"} [30d:5m]) /node_memory_MemTotal_bytes{idc=\"$idc\",system=\"$system\"}",
      "legendFormat": "",
      "interval": "",
      "exemplar": false,
      "datasource": {
        "type": "prometheus",
        "uid": "k3SQ5N_Gk"
      },
      "format": "table",
      "instant": true,
      "refId": "A"
    },
    {
      "expr": "100- 100* avg_over_time(node_memory_MemAvailable_bytes{idc=\"$idc\",system=\"$system\"} [30d:10m]) /node_memory_MemTotal_bytes{idc=\"$idc\",system=\"$system\"}",
      "legendFormat": "",
      "interval": "",
      "exemplar": false,
      "datasource": {
        "type": "prometheus",
        "uid": "k3SQ5N_Gk"
      },
      "format": "table",
      "hide": false,
      "instant": true,
      "refId": "B"
    },
    {
      "expr": "100 - min_over_time(avg by(instance,system,idc,environment,instancename) (rate(node_cpu_seconds_total{mode=\"idle\",idc=\"$idc\",system=\"$system\"}[2m]))[30d:5m]) * 100",
      "legendFormat": "",
      "interval": "",
      "exemplar": false,
      "datasource": {
        "type": "prometheus",
        "uid": "k3SQ5N_Gk"
      },
      "format": "table",
      "hide": false,
      "instant": true,
      "refId": "C"
    },
    {
      "expr": "100 - avg_over_time(avg by(instance,system,idc,environment,instancename) (rate(node_cpu_seconds_total{mode=\"idle\",idc=\"$idc\",system=\"$system\"}[2m]))[30d:5m]) * 100",
      "legendFormat": "",
      "interval": "",
      "exemplar": false,
      "datasource": {
        "type": "prometheus",
        "uid": "k3SQ5N_Gk"
      },
      "format": "table",
      "hide": false,
      "instant": true,
      "refId": "D"
    }
  ],
  "datasource": null
}

tpl:

=== 1:
include::grafana_panel_query_table:11[query="A",dashboard="wVCbu81nz",var-DS_PROMETHEUS="Prometheus-tencent",job="consul-node-exporter",idc="tencent-cloud",system="k8s-2b",environment="pro"]

error log:

I, [2022-03-23T10:02:12.831453 #52012]  INFO -- : You are running ruby-grafana-reporter version 0.5.2.
D, [2022-03-23T10:02:12.832068 #52012] DEBUG -- : Document attributes: {"convert-backend"=>"pdf", "imagesdir"=>"./images", "var-template"=>"b"}
D, [2022-03-23T10:02:12.832504 #52012] DEBUG -- : Template ./templates/b.adoc contains 1 calls of grafana reporter functions.
D, [2022-03-23T10:02:12.834195 #52012] DEBUG -- : Processing PanelQueryTableIncludeProcessor (instance: default, dashboard: wVCbu81nz, panel: 11, query: A)
D, [2022-03-23T10:02:12.834410 #52012] DEBUG -- : Requesting https://monitor.sxftwork.com/api/frontend/settings with '' and timeout '60'
D, [2022-03-23T10:02:15.980700 #52012] DEBUG -- : Received response #<Net::HTTPOK:0x00000000085c5690>
D, [2022-03-23T10:02:15.982624 #52012] DEBUG -- : Requesting https://monitor.sxftwork.com/api/dashboards/uid/wVCbu81nz with '' and timeout '60'
D, [2022-03-23T10:02:16.263501 #52012] DEBUG -- : Received response #<Net::HTTPOK:0x0000000008437788>
D, [2022-03-23T10:02:16.266698 #52012] DEBUG -- : Requesting https://monitor.sxftwork.com/api/datasources/proxy/1/api/v1/query?time=1648000931000&query=100-+100%2A+min_over_time%28node_memory_MemAvailable_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D+%5B30d%3A5m%5D%29+%2Fnode_memory_MemTotal_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D with '' and timeout '60'
D, [2022-03-23T10:02:16.496500 #52012] DEBUG -- : Received response #<Net::HTTPOK:0x00000000083c6dd0>
E, [2022-03-23T10:02:16.498165 #52012] ERROR -- : GrafanaReporterError: The datasource request to 'Prometheus-tencent' (Grafana::PrometheusDatasource) failed with an internal error: undefined method `[]' for nil:NilClass
I, [2022-03-23T10:02:16.581694 #52012]  INFO -- : Report creation ended after 4 seconds with status 'finished'
divinity666 commented 2 years ago

Unfortunately the recent changes did not improve the debugging here. I just released a new version, please reevaluate and share the error log, which should be definitely a bit more explicit.

myselfghost commented 2 years ago

Unfortunately the recent changes did not improve the debugging here. I just released a new version, please reevaluate and share the error log, which should be definitely a bit more explicit.

I, [2022-03-24T15:31:19.283911 #60256]  INFO -- : Report started at 2022-03-24 15:31:19 +0800
I, [2022-03-24T15:31:19.284770 #60256]  INFO -- : You are running ruby-grafana-reporter version 0.5.3.
D, [2022-03-24T15:31:19.285466 #60256] DEBUG -- : Document attributes: {"convert-backend"=>"pdf", "imagesdir"=>"./images", "var-template"=>"b"}
D, [2022-03-24T15:31:19.285797 #60256] DEBUG -- : Template ./templates/b.adoc contains 1 calls of grafana reporter functions.
D, [2022-03-24T15:31:19.287312 #60256] DEBUG -- : Processing PanelQueryTableIncludeProcessor (instance: default, dashboard: wVCbu81nz, panel: 11, query: A)
D, [2022-03-24T15:31:19.287522 #60256] DEBUG -- : Requesting https://monitor.sxftwork.com/api/frontend/settings with '' and timeout '60'
D, [2022-03-24T15:31:21.796484 #60256] DEBUG -- : Received response #<Net::HTTPOK:0x00000000083f09c8>
D, [2022-03-24T15:31:21.798269 #60256] DEBUG -- : Requesting https://monitor.sxftwork.com/api/dashboards/uid/wVCbu81nz with '' and timeout '60'
D, [2022-03-24T15:31:22.053655 #60256] DEBUG -- : Received response #<Net::HTTPOK:0x0000000008251bf8>
F, [2022-03-24T15:31:22.057681 #60256] FATAL -- : undefined method `gsub' for nil:NilClass
I, [2022-03-24T15:31:22.140242 #60256]  INFO -- : Report creation ended after 3 seconds with status 'finished'
divinity666 commented 2 years ago

Ok, I see where this bug is coming from. Can you confirm, that the datasource Prometheus-tencent is the datasource used for this panel queries?

myselfghost commented 2 years ago

Ok, I see where this bug is coming from. Can you confirm, that the datasource Prometheus-tencent is the datasource used for this panel queries?

yes

divinity666 commented 2 years ago

Ok, I again released a version, fixing the recent bug. Please test again. I now expect to get the needed debug information.

myselfghost commented 2 years ago

Ok, I again released a version, fixing the recent bug. Please test again. I now expect to get the needed debug information.

I, [2022-03-25T14:56:58.655545 #5512]  INFO -- : You are running ruby-grafana-reporter version 0.5.4.
D, [2022-03-25T14:56:58.656202 #5512] DEBUG -- : Document attributes: {"convert-backend"=>"pdf", "imagesdir"=>"./images", "var-template"=>"b"}
D, [2022-03-25T14:56:58.656542 #5512] DEBUG -- : Template ./templates/b.adoc contains 1 calls of grafana reporter functions.
D, [2022-03-25T14:56:58.658149 #5512] DEBUG -- : Processing PanelQueryTableIncludeProcessor (instance: default, dashboard: wVCbu81nz, panel: 11, query: A)
D, [2022-03-25T14:56:58.658400 #5512] DEBUG -- : Requesting https://monitor.sxftwork.com/api/frontend/settings with '' and timeout '60'
D, [2022-03-25T14:57:00.906773 #5512] DEBUG -- : Received response #<Net::HTTPOK:0x000000000855fc50>
D, [2022-03-25T14:57:00.910070 #5512] DEBUG -- : Requesting https://monitor.sxftwork.com/api/dashboards/uid/wVCbu81nz with '' and timeout '60'
D, [2022-03-25T14:57:01.172216 #5512] DEBUG -- : Received response #<Net::HTTPOK:0x00000000083f60a8>
D, [2022-03-25T14:57:01.175805 #5512] DEBUG -- : Requesting https://monitor.sxftwork.com/api/datasources/proxy/1/api/v1/query?time=1648191417000&query=100-+100%2A+min_over_time%28node_memory_MemAvailable_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D+%5B30d%3A5m%5D%29+%2Fnode_memory_MemTotal_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D with '' and timeout '60'
D, [2022-03-25T14:57:01.438524 #5512] DEBUG -- : Received response #<Net::HTTPOK:0x000000000833d800>
E, [2022-03-25T14:57:01.439728 #5512] ERROR -- : GrafanaError: The datasource request returned with an unsupported response format (received: {"status":"success","data":{"resultType":"vector","result":[]}}). (Grafana::UnsupportedQueryResponseReceivedError)
I, [2022-03-25T14:57:01.548180 #5512]  INFO -- : Report creation ended after 3 seconds with status 'finished'
divinity666 commented 2 years ago

Which grafana version are you using?

myselfghost commented 2 years ago

Which grafana version are you using?

v8.3.5

divinity666 commented 2 years ago

Ok, looks as if Grafana modified the datasource query methods for Prometheus in Release 8.3.4, refer release notes.

I need to check on how this could fixed.

If you have the option to try out with Grafana 8.3.3 or earlier, it would be great to see, if it is working there.

divinity666 commented 2 years ago

Ok, after a few tests, it looks like the old API should still be working properly.

I now checked some more things in your logs, which do seem odd to me:

1. Your template call (optional TODO) In your initial message you posted the following template call:

    include::grafana_panel_query_table:11[query="A",dashboard="wVCbu81nz",var-DS_PROMETHEUS="Prometheus-tencent",job="consul-node-exporter",idc="tencent-cloud",system="k8s-2b",environment="pro"]

and the following HTTP call as a result in the logs:

https://monitor.sxftwork.com/api/datasources/proxy/1/api/v1/query?time=1648000931000&query=100-+100%2A+min_over_time%28node_memory_MemAvailable_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D+%5B30d%3A5m%5D%29+%2Fnode_memory_MemTotal_bytes%7Bidc%3D%22tencent-cloud%22%2Csystem%3D%22devops%22%7D

a) Why have the variables in your HTTP call be replaced. This should only be possible, if the variables in your template call would be prefixed with var-, so the template call should look like that:

include::grafana_panel_query_table:11[query="A",dashboard="wVCbu81nz",var-DS_PROMETHEUS="Prometheus-tencent",var-job="consul-node-exporter",var-idc="tencent-cloud",var-system="k8s-2b",var-environment="pro"]

b) Somehow the variable system in the HTTP call is set to devops instead of k8s-2b.

Maybe you could try the corrected template call here, so that we get some consistency between template call and HTTP response.

2. The time variable (most likely the fix) For some reason, time internally handed over as msecs to Prometheus, though it would expect it as secs. If this is the issue, the following call, should be working fine:

include::grafana_panel_query_table:11[query="A",dashboard="wVCbu81nz",var-DS_PROMETHEUS="Prometheus-tencent",var-job="consul-node-exporter",var-idc="tencent-cloud",var-system="k8s-2b",var-environment="pro",to="1648000931"]

Looking forward for your feedback

divinity666 commented 1 year ago

This case is related to #28 and exists in conjunction with grafana >=7.5 for prometheus datasources.

divinity666 commented 1 year ago

Please try again with the just released version, which should fix the issue. Please close the issue, if it is working as expected.

divinity666 commented 1 year ago

Closing because of missing input