search

grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
65.19k stars 12.16k forks source link

Data sources: Unable to add a Loki data source in Grafana v11.3.0 #96288

Closed sovereignstack closed 1 week ago

sovereignstack commented 1 week ago

What happened?

I am getting an error while adding a Loki datasource.

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=checkHealth pluginId=loki dsName=loki-2 dsUID=ae3pq986ypudcd uname=admin fromAlert=false t=2024-11-12T10:57:40.238122451Z level=error msg="Error received from Loki" duration=4.935173ms stage=databaseRequest statusCode=400 contentLength=65 start=1970-01-01T00:00:01Z end=1970-01-01T00:00:04Z step=1s query=vector(1)+vector(1) queryType=instant direction=backward maxLines=0 supportingQueryType=none lokiHost=loki-stack.monitor.svc.cluster.local:3100 lokiPath=/loki/api/v1/query status=error error="parse error at line 1, col 1: syntax error: unexpected IDENTIFIER" statusSource=downstream

logger=tsdb.loki endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth endpoint=CheckHealth t=2024-11-12T10:57:40.238219665Z level=error msg="Loki health check failed" error="error from loki: parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"

Image

Image

There is no problem in network permissions, DNS resolution or port number Loki server is reachable from grafana pod as seen from the curl request. Image

What did you expect to happen?

Expected a success message while saving the data source

Did this work before?

It is working in previous version of Grafana installed in the cluster i.e. Grafana [v9.3.0 (e9cb2a313e)]

How do we reproduce it?

  1. Run grafana instance using this container image docker.io/grafana/grafana@sha256:a0f881232a6fb71a0554a47d0fe2203b6888fe77f4cefb7ea62bed7eb54e13c3
  2. Try to add a Loki data source running on non standard port i.e. if using http use port 3100.
  3. Click on Save & Test

Is the bug inside a dashboard panel?

No

Environment (with versions)?

Grafana: v11.3.0 (d9455ff7db) Container Image OS: CentOS Linux 7 (Core) Browser: Chrome 130.0.6723.117 (Official Build) (64-bit)

Grafana platform?

Kubernetes

Datasource(s)?

Loki 2.6.1 Loki was installed from a helm chart

chart: loki-stack
repoURL: https://grafana.github.io/helm-charts
targetRevision: 2.8.7
svennergr commented 1 week ago

Hi @sovereignstack,

the problem is that you are using quite an old Loki with version 2.6.1. We use a special query containing the vector(1) expression as a health check, and thus this health check will fail with Loki versions that do not support vector(1).

AFAIK the helm charts, also the one at https://github.com/grafana/helm-charts/tree/main/charts/loki, should offer newer Loki versions, so I suggest upgrading. Nevertheless, even if the health check fails and adding a data source shows an error, you should be able to use it in Explore or Dashboards after adding.