[grafana] Data sources are not loaded properly when Grafana is configured with anonymous log in

[mihail-i4v]

Hi, we've recently upgraded Grafana from 8.3.1 to 8.3.3 (helm chart versions 6.18.2 to 6.20.5) and noticed that the data sources are not loading as expected.

What we can see in the logs for the grafana-sc-datasources sidecar:

Starting collector
No folder annotation was provided, defaulting to k8s-sidecar-target-directory
Selected resource type: ('secret', 'configmap')
Loading incluster config ...
Config for cluster api at 'https://192.168.176.1:443' loaded...
Unique filenames will not be enforced.
5xx response content will not be enabled.
Working on ADDED configmap monitoring/grafana-datasource-prometheus
POST request sent to http://localhost:3000/api/admin/provisioning/datasources/reload. Response: 403 Forbidden {
  "message": "Permission denied"
}

Working on ADDED configmap monitoring/grafana-datasource-bigquery
POST request sent to http://localhost:3000/api/admin/provisioning/datasources/reload. Response: 403 Forbidden {
  "message": "Permission denied"
}

I suspect this is happing because the Admin HTTP API request basic authentication to be enabled whereas on our Grafana we've disabled this.

It seems the REQ_URL and REQ_METHOD are always specified starting from this commit(https://github.com/grafana/helm-charts/commit/ee0914e19e680a6cde7a5a4e9bcb43f2bef9517a). If I remove these two parameters, everything works as expected.

Should we introduce a variable in the helm chart that removes these two variables similar to the GF_SECURITY_ADMIN_USER and GF_SECURITY_ADMIN_PASSWORD which leave out the REQ_USERNAME and REQ_PASSWORD?

I'm happy to raise a PR for this if you agree is the way forward.

[gasmick]

why do we need this sidecar if reload is skipped? I have the same situation - basic authentication is disabled, thus there is no access to the admin HTTP API. But I want the datasources to be up to date without needing to restart the grafana container in the pod. As I understand your proposed solution requires the pod to be replaced, cause the loading is then done only in an initContainer. Or am I wrong?

[ruiengana]

If I keep both Basic Authentication and Anonymous Authentication enabled I don't see the reload permission error and I can still open Grafana anonymously.

[c4tz]

This problem (with another HTTP status code) seems to also appear when specifying GF_SECURITY_ADMIN_USER and GF_SECURITY_ADMIN_PASSWORD in .Values.env.

By doing so, REQ_USERNAME and REQ_PASSWORD are left out for the sidecar container, making the reload call fail with:

POST request sent to http://localhost:3000/api/admin/provisioning/datasources/reload. Response: 401 Unauthorized {"message":"Unauthorized"}

I wanted to use akv2k8s' env variable injection to set the admin password, but this doesn't seem to be possible at the moment due to this bug.

[on3iropolos]

For me, the problem was that the sample file (which I used) located at: https://github.com/grafana/helm-charts/tree/main/charts/grafana#sidecar-for-datasources. This file example is using yaml without values. Ideally this configuration example should include key: {} or key: []. My troubleshooting steps:

1. Connected to sidecar shell, ran echo $REQ_USERNAME $REQ_PASSWORD
2. Ran wget --post-data "" --server-response --spider --timeout 10 --tries 1 http://username:password@localhost:3000/api/admin/provisioning/datasources/reload. Monitored grafana container logs.
3. Verified my sidecar was enabled: true and configured to hit: url: http://localhost:3000/api/admin/provisioning/datasources/reload
4. Verified configmaps were loaded correctly in Rancher / Kubernetes.
5. Checked logs of sidecar container. Saw HTTP 500 errors.
6. Checked logs of grafana container. Saw invalid yaml errors for the API reload.

[y0zg]

Any plans to fix this bug?

[maxpain]

The same problem

[dbluxo]

Oh no, we get this error too

[radoslav-stefanov]

Same problem here. However as per @ruiengana comment enabling basic auth and anonymous login is enough to workaround it.