Response "Unauthorized"

[ScubaDrew]

Issue details

I followed the steps to install and setup the proxy as detailed in the README.md

When I point my webbrowser at the proxy address I just get back an "unauthorized" response.

Environment

Dashboard version: latest as of 5/28/16 Kubernetes version: 1.2.2 Operating system: core

Steps to reproduce

1. setup cluster on AWS per https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html
2. install dashboard as described in README.md
3. I have an ubuntu server that I run kubectl proxy --address="172.20.32.54" --port=9090
4. point webbrowser on osx system at the ubuntu server address (...32.54 address above), get successful http connection but response is simply "unauthorized"

All other kubectl commands work correctly.

kubectl cluster-info Kubernetes master is running at https://kub2.drewoconnor.com Heapster is running at https://kub2.drewoconnor.com/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://kub2.drewoconnor.com/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://kub2.drewoconnor.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

Observed result

Unauthorized

Expected result

Dashboard

[bryk]

Can you kubectl get pods --namespace=kube-system and kubectl logs <pod-of-dashboard> --namespace=kube-system?

[ScubaDrew]

Hi bryk, I found the issue here.

I was using the kubectl proxy command as noted above on an ubuntu server, specifying the IP address in the command. I was when pointing a browser on an OSX machine at the ubuntu server's IP. The connection worked, and I got the unauthorized response. I found that if i used "kubectl --port=9090" and then did a wget to localhost, the request worked as it should.

This appears to be an issue with the proxy command in kubectl. Or perhaps they intended to limit connections to localhost.

I don't believe this is an issue with the web ui.

Thanks, Drew

[bryk]

@ScubaDrew Yeah, that's what I expected. I'm closing this issue. Please reopen if needed.

[EamonZhang]

I have the same issue .

$kubectl proxy --port=9090
Starting to serve on 127.0.0.1:9090

curl 127.0.0.1:9090
<h3>Unauthorized</h3>

curl 127.0.0.1:9090/ui
<a href="/ui/">Moved Permanently</a>.

What can i do, thanks

[bryk]

Yeah, that's correct that it is moved permanently. Can you open the URL with a browser?

[EamonZhang]

@bryk

$kubectl proxy --address="10.2.0.10"  --port=9090
Starting to serve on 10.2.0.10:9090

or

$kubectl proxy --address="0.0.0.0"  --port=9090
Starting to serve on 0.0.0.0:9090

in browser window

http://10.2.0.10:9090/ui

Unauthorized

[ScubaDrew]

@EamonZhang it only works when the url is localhost. Accessing it via IP address is... Unauthorized by design.

[EamonZhang]

@ScubaDrew

Server computer have no browser to be used. Does nginx proxy support . or other measures ?

Thanks

[bryk]

If your master is publicly accessible you can access it to see the UI: https://master/ui or your clients can use kubectl proxy on their machines. Finally, you can expose the UI as external service and access it from outside world.

[hwinkel]

I have the master running on a public IP but get unauthorized un http://ip/ui

the dashboard pod self is not running on the master node. But another one. kubectl cluster-info does not show the kubernetes-dashboard. But the service is listed and the pod as well

How do I expose the UI? any hints?

[natejoebott]

I have the same issue as hwinkel above. Just installed K8 and the dashboard per https://github.com/kubernetes/dashboard#kubernetes-dashboard

I get the 'unauthorized" message when accessing https:///ui

I am using a mozilla browser from a windows client, so the kubectl proxy approach doesn't seem appropriate. What I am missing here?

[remonlam]

Same here running 1.4 installed it following the guide at http://kubernetes.io/docs/getting-started-guides/kubeadm/.

However when installing the dashboard/UI it seems to be running but get an "Unauthorized"

Followed the guide at http://kubernetes.io/docs/user-guide/ui/ to install the UI basically just run "kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml"

kubectl get pods --all-namespaces

NAMESPACE     NAME                                         READY     STATUS    RESTARTS   AGE
kube-system   etcd-kube-master.net.loc                      1/1       Running   1          19m
kube-system   kube-apiserver-kube-master.net.loc            1/1       Running   1          20m
kube-system   kube-controller-manager-kube-master.net.loc   1/1       Running   1          20m
kube-system   kube-discovery-982812725-s79yq                1/1       Running   1          20m
kube-system   kube-dns-2247936740-b9a2o                     3/3       Running   3          20m
kube-system   kube-proxy-amd64-4p9vg                        1/1       Running   1          20m
kube-system   kube-proxy-amd64-cbbrm                        1/1       Running   0          20m
kube-system   kube-proxy-amd64-ela05                        1/1       Running   0          20m
kube-system   kube-scheduler-kube-master.net.loc            1/1       Running   1          19m
kube-system   kubernetes-dashboard-1655269645-arfpw         1/1       Running   0          15m
kube-system   weave-net-cvcgd                               2/2       Running   0          19m
kube-system   weave-net-hxkwf                               2/2       Running   2          19m
kube-system   weave-net-pwuto                               2/2       Running   0          19m

@natejoebott are you running 1.4?

[natejoebott]

Right, I too installed it via the beta version of Kubeadm with 1.4. Upon further investigation this may be expected behavior based upon the limitation number 4 - There is not yet an easy way to generate a kubeconfig file which can be used to authenticate to the cluster remotely with kubectl.

[colemickens]

Are people expecting the dashboard to be publicly available without auth? If this were really the case, everyone would be exposing write access for their cluster to the anonymous world. Unless I'm missing a detail here?

[natejoebott]

No, but there doesn't seem to be simple way to auth. In the documentation for the dashboard, it provides the follow guidance after installation: And then navigate to https:///ui If it asks password, use $ kubectl config view to find it.

I was not prompted for a password, nor does the kubectl config view provide any indication of a password. Next up is using the alternative proxy method; however, only localhost can be used - http://localhost:8001/ui - thus unless you are using a mac or linux host where the kubectl proxy script was invoked will this work.

[colemickens]

Presumably you're using client-cert auth then if there's no password/token in kubeconfig. You can configure your browser to send the relevant client certificate, but it's usually not-straightforward in my experience.

kubectl proxy is certainly easier. There is a build of kubectl for Windows and if you want, you can run it with flags that enable it to listen on all interfaces and for connections from all hosts so that you can run it on a server machine and allow anyone to hit it (this is obviously insecure).

(I'm working on an example of how you can run a reverse-proxy with auth in front of the dashboard that should make things like this easier.)

[bryk]

(I'm working on an example of how you can run a reverse-proxy with auth in front of the dashboard that should make things like this easier.)

Can you share any details here? I'd love us to incorporate something like this to default install, so that folks can expose the UI to external world.

[colemickens]

This is the idea: https://github.com/kubernetes/contrib/issues/1492. I still haven't had time to put the oauth2_proxy configuration together. Doesn't really solve for out-of-the-box though, as oauth2_proxy will require a configmap/secret with oauth2 secrets to work the way I'm imagining.

[bryk]

All right. Share anything you make work :) We need to explore all possible solutions, because, eventually, we need to bake a solution to this into Dashboard.

[dannyman]

Heya, so we are all actively commenting on an issue that is closed since May .. begs the question as to whether this issue is Closed? (Maybe a scope change...)

Here's where we are at, total newbs: 1) Install Kubernetes with fancy new kubeadm tool: http://kubernetes.io/docs/getting-started-guides/kubeadm/ 2) Per above "Explore other add-ons" ... http://kubernetes.io/docs/admin/addons/ 3) "Dashboard is a dashboard web interface for Kubernetes." -- wow this sounds useful for us newbs! 4) https://github.com/kubernetes/dashboard#kubernetes-dashboard 5) kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml 6) https:///ui --> Unauthorized 7)

root@kub-test0:~# kubectl config view
apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

8) Find this issue and:

root@kub-test0:~# wget --no-check-certificate https://localhost/ui
--2016-10-04 15:28:58--  https://localhost/ui
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:443... connected.
WARNING: cannot verify localhost's certificate, issued by ‘CN=kubernetes’:
  Unable to locally verify the issuer's authority.
WARNING: no certificate subject alternative name matches
    requested host name ‘localhost’.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

So since this is my first time on a test cluster I can do this: kubectl proxy --address 0.0.0.0 Nope: <h3>Unauthorized</h3>

Maybe helpful would be a pointer on how to add a username/password in the Usage section at https://github.com/kubernetes/dashboard#kubernetes-dashboard

[colemickens]

What URL did you try to hit after running kubectl proxy?

[dannyman]

From my workstation, to external IP of kubernetes cluster Master: http://10.10.1.188:8001/ --> <h3>Unauthorized</h3>

From localhost on kubernetes cluster Master: https://localhost/ui --> 401 Unauthorized Username/Password Authentication Failed.

[colemickens]

And what happens if you try, after running kubectl proxy of course:

• http://10.10.1.188:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all
• http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all

[dannyman]

Workstation -> http://10.10.1.188:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all -> Unauthorized

Kubenetes Master wget http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all I get an index.html that informs me that I am using an outdated browser.

[anutech2001]

@dannyman I got the same problem. So, i try using NodePort and it's work. I can use dashboard right now on the browser. My env.

• windows 7, vagrant + centos 7, kubernetes 1.4

[bbalzola]

kubectl describe services kubernetes-dashboard --namespace=kube-system

Use the NodePort work for me on Linux

[bryk]

@anutech2001 @bbalzola Can you contribute updates to our installation guide or troubleshooting section? I'd love us to have less issues when installing on home-made clusters.

[dannyman]

Edit: even easier, as @bbalzola points out two comments above, you can bypass all the weird hinky "security" "Unauthorized" restrictions by hitting the NodePort directy: kubectl describe services kubernetes-dashboard --namespace=kube-system

You can stop reading about the days of toil we have spent figuring out how to get to the web admin ... now, we have different problems in that we have exposed our web admin interface to all comers, and really the challenge is we have to figure out secure access to NodePorts.

Question: is direct, unauthenticated access to the app via the NodePort a best practice in Kubenetes apps or is this feature actually a bug?

---

Yes I have two work-arounds figured out:

• Install a modern web browser on your Kubenetes host and forward it to your workstation via X11 OR
• Install Kubenetes on your local workstation and copy certificates over from your Kubenetes host Then, on whatever host you want to run your web browser: kubectl proxy And visit: http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all

Our verdict is that the web admin appears to be very nice not only for interpreting cluster state, but for deploying services, &c. Thus, we should try to support it for our cluster users, so we have a research spike to figure out Kubenetes cluster user authentication and access control. Unfortunately, that is a big hot nasty mess: http://kubernetes.io/docs/admin/authentication/

There is a bit of a chicken and egg problem here, in that web admin is a user-friendly tool for new users, but you can not really use it unless you are clever enough to configure one of the trickier, murkier parts of Kubernetes. If you are new, then you probably want to "get kubernetes running on your workstation and copy over the certificates used on the cluster" then you can get the proxy back to the cluster working. (I have not done that part but I reckon there's a good link that covers copying the certificates over in a basic setup, and that the web admin docs could offer that link to help people past this issue.)

[bryk]

Question: is direct, unauthenticated access to the app via the NodePort a best practice in Kubenetes apps or is this feature actually a bug?

This is more like a known limitation. We want to solve this problem but so far haven't had resources to do so. @colemickens Had done some research on putting an oauth proxy next to the UI.

[groyee]

I am sorry, I got lost here.

Is there any way or any workaround to work with the ui dashboard?

root@kubserver1:~# kubectl proxy Starting to serve on 127.0.0.1:8001

root@kubserver1:~$ curl http://localhost:8001/ Unauthorized

I see that people here complain about this issue since April. If your documentation is wrong then why don't you update it?

http://kubernetes.io/docs/user-guide/ui/

[seeekr]

@groyee quoting from above:

kubectl describe services kubernetes-dashboard --namespace=kube-system

Then just connect to the NodePort that is given in the output and it works. (This also means that everyone can potentially use your dashboard if they happen to find the right port, because it doesn't do auth.)

[groyee]

Doesn't work:

I tried:

1. http://localhost:31745 from localhost. No response, curl just waits for response.
2. http://public-ip:31745 from remote host using Chrome. Same thing.
3. ssh tunneling port 31745. No go

[seeekr]

@groyee Ok. Option 2 definitely works for me, using kube 1.4, installed using kubeadm. Being far from an expert on k8s, my wisdom here stops at: Check, in some way, that both the cluster as well as the UI services are completely up and running? "curl just waits for response" sounds like you're on the right track, but the service listening isn't operational or something.

[mcapuccini]

@groyee please check the wiki of my ansible/terraform deployment https://github.com/mcapuccini/KubeNow/

I find quite convenient to use ssh tunnelling.

[groyee]

It looks like the tutorial is only for OpenStack. I am using Azure.

[colemickens]

How did you bring it up on Azure? If you used azkube or kubernetes-anywhere then I'd recommend using the kubeconfig along with kubectl proxy. Should just work.

[mcapuccini]

@groyee we will add GCE, and AWS soon, but not Azure I am afraid (but if you'd like to contribute I'll be happy to merge PRs, it's quite simple to make a terraform module). Said that, you can just give a look to this ansible playbooks, and run the commands manually in your deployment:

On master: https://github.com/mcapuccini/KubeNow/blob/master/playbooks/roles/proxy-daemon/tasks/main.yml

On your local machine: https://github.com/mcapuccini/KubeNow/blob/master/playbooks/ui-tunnels-add.yml

then you just browse: http://localhost:8001/ui

[groyee]

Thank you guys.

I uninstalled 1.4.1, installed a clean 1.4 and after that the NodePort works for me.

Life is not so easy with Kubernetes :-)

[jimjsong]

newb here. I followed the comments and instructions, and I was able to get my kubernetes dashboard to show up..yay...thank you all!!

I installed the monitoring and logging via http://kubernetes.io/docs/user-guide/monitoring/ and specifically grafana and influxdb (https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md#metrics). I can't seem to get this to pull up on the NODEPORT no matter what combinations of url manipulations I try. I finally installed a desktopo gui on my linux server and I was able see grafana but only via "localhost". Any way to have grafana show up on the real ip address so it is accessible without having to log into the MASTER? THANKS ALL!

[iamsortiz]

In my case, messing around with the getting started kubernetes-on-vagrant-single.

I went the ssh port forwarding way. Here is the gist to automate it.

• It asks for the password, to automate that you can use ssh-pass as seen in stackoverflow
  • or go nuclear and use proper SSH credentials

https://gist.github.com/iamsortiz/9b802caf7d37f678e1be18a232c3cc08

#!/bin/bash

# Usage: Assuming a vagrant based kubernetes (as in https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html), run this script in the same folder of the Vagrantfile (where you would normally do "vagrant up")
# * Then insert the password (by default: kubernetes)
# * Browse localhost:9090

USERNAME='kubernetes'
PASSWORD='kubernetes'

function main() {
  Create_user_on_kubernetes_machine

  SSH_port_forwarding

  # Enjoy (at localhost:9090)
}

function Create_user_on_kubernetes_machine() {
  vagrant ssh -c "if [ ! -d /home/$USERNAME ]; then sudo useradd $USERNAME -m -s /bin/bash && echo '$USERNAME:$PASSWORD' | sudo chpasswd; fi"
}

function SSH_port_forwarding() {
  KUBERNETES_HOST=$(kubectl cluster-info | head -n 1 | grep -o -E '([0-9]+\.){3}[0-9]+')
  TARGET=$(kubectl describe services kubernetes-dashboard --namespace=kube-system | grep Endpoints | awk '{ print $2 }')

  ssh -L 9090:$TARGET $USERNAME@$KUBERNETES_HOST
}

main

[faddat]

Same trouble here. I'm going to try the solutions described. I was using kubeadm to set up the cluster like the other users reporting the issue.

Is anyone able to tell me what exactly the issue is here? I realize there's probably alreadyagreement on this but I'm just not too sure what it is yet. I will try using caddy to RP the dashboard.

[groyee]

It didn't work for me as well.

What @seeekr suggested worked for me.

[bryk]

Same trouble here. I'm going to try the solutions described. I was using kubeadm to set up the cluster like the other users reporting the issue.

It seems that kubeadm tool and its networking are doing something wrong. Can you report an issue there and link to this one?

If you want to try out the UI, use minikube. It works and has dashboard out-of-the-box.

[Starefossen]

Having similar problem on a cluster set up by kubeadm:

$ kubectl get po -n kube-system

NAME                                                                      READY     STATUS             RESTARTS   AGE
kubernetes-dashboard-1655269645-ltgvv                                     0/1       CrashLoopBackOff   13         47m

$ kubectl logs kubernetes-dashboard-1655269645-ltgvv -n kube-system

Starting HTTP server on port 9090
Creating API server client for https://10.0.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.0.0.1:443/version: dial tcp 10.0.0.1:443: getsockopt: no route to host

$ curl https://10.0.0.1:443/version
curl: (60) Peer's Certificate issuer is not recognized.

$ curl -k https://10.0.0.1:443/version
Unauthorized

I suspect kubeadm is not configuring any unauthenticated kube-apiserver service endpoints 😕

$ kubectl describe svc kubernetes
Name:                   kubernetes
Namespace:              default
Labels:                 component=apiserver
                        provider=kubernetes
Selector:               <none>
Type:                   ClusterIP
IP:                     10.0.0.1
Port:                   https   443/TCP
Endpoints:              172.18.0.134:6443
Session Affinity:       ClientIP

[cheld]

I have not yet tried kubeadm myself. Maybe your problem is described in the shooting guide: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

Could you please check?

[faddat]

Checking. Thanks!

[asheshambasta]

+1. Using NodePort results in a 404 not found for me.

[faddat]

My update is that I've tried what I know to try and what the docs suggest trying without avail. I don't think it's broken exactly, because I can see the dashboard running there. That said I cannot connetct to it.

Did the location of the CA Certs move? Are they now directly in /etc/kuberenetes? I can't find CA Certs in /etc/kubernetes/certs where I recall reading they'd be.

[asheshambasta]

@faddat the output of the setup usually dumps;

<master/pki> created keys and certificates in "/etc/kubernetes/pki"

[asheshambasta]

I must say this is quite a dead end for me. http://<ip>:<NodePort>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=_all fails to load on my browser and instead tries to download files that fail to decompress.