Services "monitoring-grafana" is forbidden: User "system:anonymous" cannot get services/proxy in the namespace "kube-system"

[zedtux]

Description

I have deployed the k8s dashboard and saw errors from k8s (using kail) stating that heapster was missing.

$ kubectrl cluster-info
Kubernetes master is running at https://XX.XX.XXX.XXX:6443
Heapster is running at https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
monitoring-grafana is running at https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
monitoring-influxdb is running at https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

I started the kubectrl proxy command and tried to access https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy but I have the following output:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "services \"monitoring-grafana\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "monitoring-grafana",
    "kind": "services"
  },
  "code": 403
}

Sorry I'm new to k8s and heapster.

Steps to reproduce:

I installed heapster following the InfluxDB instructions :

$ kubectl create -f deploy/kube-config/influxdb/
$ kubectl create -f deploy/kube-config/rbac/heapster-rbac.yaml
$ kubectrl proxy

Using a web browser go to https://XX.XX.XXX.XXX:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

Describe the results you received:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "services \"monitoring-grafana\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "monitoring-grafana",
    "kind": "services"
  },
  "code": 403
}

Describe the results you expected:

Grafana web UI

Output of heapster --version:

Don't know where to run heapster --version ... but the Docker image used is k8s.gcr.io/heapster-grafana-amd64:v4.4.3.

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

[DirectXMan12]

You don't have RBAC rules set up to allow unauthenticated users to proxy to the grafana service. Please read up on RBAC in the official Kubernetes documentation. You'll need to allow anonymous user GET the services/proxy resource for the grafana service.

[zedtux]

Thank you @DirectXMan12 for closing my issue .. but as I stated in my issue, I followed your wiki and got this issue.

Unless I missed something in the wiki (in this case, please point me to it) otherwise the wiki should be updated.

I'm new to k8s and your comment is not helping me (I mean I was able to understand that I have an RBAC issue with unauthenticated users). This issue is more like to improve the project's doc in order to help new comers like me.

[DirectXMan12]

I apologize for hastily closing this issue, I did not intend to be rude. I often get 40-100 new/updated issues a day that I have to triage, and leaving issues open after I think they've been answered just makes triage later more difficult. Please feel free to reopen issues if you think they were closed incorrectly.

On the subject of docs updates, can one of the influxdb owners (@andyxning @acobaugh) please update the docs to mention the trade offs in allowing unauthenticated access to grafana?

[zedtux]

Thank you @DirectXMan12 for your comment 👍

Am I not able to re-open the issue. There's no reopen button.

[DirectXMan12]

Am I not able to re-open the issue. There's no reopen button.

Ah, ok. Let me do that, then.

[zedtux]

@andyxning and @acobaugh any news on this please ?

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[zedtux]

I hate those stupid bots !

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[zedtux]

Damned ... /remove-lifecycle stale