Unable to access dashboard

[Rahul91]

Issue details

Unable to access dashboard on http://master_ip/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

Message "no endpoints available for service "kubernetes-dashboard""

I did the steps given in http://kubernetes.io/docs/user-guide/ui-access/, but still no result.

kubectl create -f cluster/addons/dashboard/dashboard-controller.yaml --namespace=kube-system
kubectl create -f cluster/addons/dashboard/dashboard-service.yaml --namespace=kube-system

#kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}

When using v0.19.3, I was able to access the dashboard.

Dashboard version: v1.0.1
Kubernetes version: v1.2.4
Operating system: Gnu-linux/Ubuntu
Node.js version: -
Go version: -

Observed result

Unable to access UI

[bryk]

Is the Dashboard UI running? Can you show me logs of the dashboard pod?

[arhided]

I got the same error.

Starting HTTP server on port 9090 Creating API server client for https://10.0.0.1:443 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.0.0.1:443/version: dial tcp 10.0.0.1:443: i/o timeout

Name: kubernetes-dashboard Namespace: kube-system Labels: k8s-app=kubernetes-dashboard,kubernetes.io/cluster-service=true Selector: k8s-app=kubernetes-dashboard Type:
ClusterIP IP: 10.0.0.55 Port: 80/TCP Endpoints: 172.17.0.5:9090 Session Affinity: None No events.

$ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.0.0.1 443/TCP 17h

[Rahul91]

@bryk Gist: https://gist.github.com/Rahul91/f443e58dd730e0571bcea6409adb5761 I am getting this error in http://master_ip/ui

Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'

I am running my master on a server with publicly accessible IP and minion on my local machine running in a local network. Is that the reason I am getting this error?

[floreks]

@Rahul91 As your master is located on external server try to provide apiserver-host parameter to the dashboard. It is commented inside the yaml file. Without providing it, dashboard tries to discover master node locally.

@arhided how are you running your cluster? Locally? AWS/GCE?

[arhided]

Kube is running locally.

~$ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.0.0.1 443/TCP 1d

I think Kube service is there and kube-dash autodiscovery also locationg it properly.

[floreks]

@arhided

Is it local-up-cluster script or kube-up?

local-up-cluster does not offer https endpoint as far as i know. It is only exposed using http protocol and autodiscovery is looking for master on port 443 which is https.

To make it work with dashboard locally you may want to use hyperkube script that we have on our repository in build folder. Other issue may be that docker daemon is not configured properly and dashboard container can not access kubernetes master.

@bryk any other ideas?

[arhided]

hack/local-up-cluster.sh Go version: go version go1.6.2 linux/amd64 +++ [0622 17:56:05] Building the toolchain targets: k8s.io/kubernetes/hack/cmd/teststale +++ [0622 17:56:05] Building go targets for linux/amd64: cmd/kubectl cmd/hyperkube +++ [0622 17:56:07] Placing binaries curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused API SERVER port is free, proceeding... Detected host and ready to start services. Doing some housekeeping first... Using GO_OUT /home/user/kubernetes/_output/local/bin/linux/amd64 Starting services now! Starting etcd etcd -addr 127.0.0.1:4001 -data-dir /tmp/tmp.hXTnA9YpPY --bind-addr 127.0.0.1:4001 >/dev/null 2>/dev/null Waiting for etcd to come up. +++ [0622 17:56:08] On try 2, etcd: : {"action":"set","node":{"key":"/_test","value":"","modifiedIndex":3,"createdIndex":3}} Waiting for apiserver to come up +++ [0622 17:56:09] On try 2, apiserver: : { "kind": "PodList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/pods", "resourceVersion": "10" }, "items": [] } Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

Logs: /tmp/kube-apiserver.log /tmp/kube-controller-manager.log /tmp/kube-proxy.log /tmp/kube-scheduler.log /tmp/kubelet.log

To start using your cluster, open up another terminal/tab and run:

cluster/kubectl.sh config set-cluster local --server=http://127.0.0.1:8080 --insecure-skip-tls-verify=true cluster/kubectl.sh config set-context local --cluster=local cluster/kubectl.sh config use-context local cluster/kubectl.sh

Http it there

http://127.0.0.1:8080/

{ "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1alpha1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/policy", "/apis/policy/v1alpha1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/healthz", "/healthz/ping", "/logs/", "/metrics", "/swaggerapi/", "/ui/", "/version" ] }

https://127.0.0.1:8080/

An error occurred during a connection to 127.0.0.1:8080. SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long)

[arhided]

Hypercube.sh - don`t change much, pretty much nothing.

gcr.io/google_containers/heapster:v1.0.2 "/heapster -port 8082"

http://127.0.0.1:8082/ 404 page not found

[Rahul91]

@floreks, @bryk I ran kubernetes-dashboard.yaml on my master server, with --apiserver-host commented and even ran by uncommenting and gave public IP of my server. Both of above didn't work.

On my minion server(running in a private network)

$ps -ef | grep kube
root     17834     1  1 14:24 ?        00:00:06 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=0.0.0.0 --api_servers=http://163.172.128.148:8080 --logtostderr=true
root     17851     1  0 14:24 ?        00:00:01 /opt/bin/kube-proxy --master=163.172.128.148:8080 --logtostderr=true

$docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED              STATUS              PORTS               NAMES
aa4d35d4898f        gcr.io/google_containers/kubernetes-dashboard-amd64:v1.1.0   "/dashboard --port=90"   About a minute ago   Up About a minute                       k8s_kubernetes-dashboard.9ae56f1c_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_d5712460
2dd21a03d9d2        gcr.io/google_containers/pause:0.8.0                         "/pause"                 About a minute ago   Up About a minute                       k8s_POD.3e8ce8bb_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_b9769263

$ docker logs aa4d35d4898f
Starting HTTP server on port 9090
Creating API server client for http://163.172.128.148:8080
Successful initial request to the apiserver, version: v1.2.4
Creating in-cluster Heapster client

On my master node(running on a server having public accessible IP)

# ps -ef | grep kube
root     16620     1  0 07:26 ?        00:00:01 /opt/bin/kube-scheduler --master=127.0.0.1:8080 --logtostderr=true --v=0
root     16621     1  0 07:26 ?        00:00:19 /opt/bin/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080 --etcd_servers=http://10.1.72.219:2379 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --secure-port=0 --service-node-port-range=30000-32767 --service_account_lookup=false
root     16623     1  0 07:26 ?        00:00:17 /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --v=0

#kubectl get rc --namespace=kube-system
NAME                   DESIRED   CURRENT   AGE
kubernetes-dashboard   1         1         19h

# kubectl get po --namespace=kube-system
NAME                         READY     STATUS    RESTARTS   AGE
kubernetes-dashboard-mc01o   1/1       Running   0          17h

#kubectl logs kubernetes-dashboard-mc01o
Error from server: pods "kubernetes-dashboard-mc01o" not found
root@onehop-for-docker-testing:~# kubectl logs kubernetes-dashboard-mc01o --namespace=kube-system
Error from server: Get https://0.0.0.0:10250/containerLogs/kube-system/kubernetes-dashboard-mc01o/kubernetes-dashboard: dial tcp 0.0.0.0:10250: connection refused

http://master_ip:8080

{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/autoscaling",
    "/apis/autoscaling/v1",
    "/apis/batch",
    "/apis/batch/v1",
    "/apis/extensions",
    "/apis/extensions/v1beta1",
    "/healthz",
    "/healthz/ping",
    "/logs/",
    "/metrics",
    "/resetMetrics",
    "/swagger-ui/",
    "/swaggerapi/",
    "/ui/",
    "/version"
  ]
}

http://master_ip:8080/api/v1/namespaces/kube-system/services/kubernetes-dashboard/

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "kubernetes-dashboard",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
    "uid": "b31b3fbf-3943-11e6-9ed6-0007cb03d000",
    "resourceVersion": "210445",
    "creationTimestamp": "2016-06-23T13:09:21Z",
    "labels": {
      "app": "kubernetes-dashboard"
    }
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 80,
        "targetPort": 9090,
        "nodePort": 31260
      }
    ],
    "selector": {
      "app": "kubernetes-dashboard"
    },
    "clusterIP": "192.168.3.63",
    "type": "NodePort",
    "sessionAffinity": "None"
  },
  "status": {
    "loadBalancer": {}
  }
}

http://master_ip:8080/api/v1/**proxy**/namespaces/kube-system/services/kubernetes-dashboard/

Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'

I am new to kubernetes, please help me resolve this issue.

[bryk]

$ docker logs aa4d35d4898f Starting HTTP server on port 9090 Creating API server client for http://163.172.128.148:8080 Successful initial request to the apiserver, version: v1.2.4 Creating in-cluster Heapster client

This means that the UI successfully connected to the apiserver. From now on it should just work. I guess this is something wrong with your cluster and proxy settings. Can you change dashboard service from "NodePort" to "LoadBalancer". This would give it an external ip address that you can access.

[cheld]

@arhided

which docker version are you using? which kubernetes version?

could you please paste the output of: kubectl run debain2 --image debian sleep 1000000 kubectl exec debain2-xxxx ls /var/run/secrets/kubernetes.io/serviceaccount

[arhided]

Hi

$ docker version Client: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:16:54 UTC 2015 OS/Arch: linux/amd64

Server: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:16:54 UTC 2015 OS/Arch: linux/amd64

http://127.0.0.1:8080/version { "major": "1", "minor": "3+", "gitVersion": "v1.3.0-alpha.4.893+2976e892a45fb2-dirty", "gitCommit": "2976e892a45fb2fdaf458af1c90040c095feb0d1", "gitTreeState": "dirty", "buildDate": "2016-06-20T14:02:32Z", "goVersion": "go1.6.2", "compiler": "gc", "platform": "linux/amd64" }

$ kubectl run debain2 --image debian sleep 1000000 deployment "debain2" created

$ kubectl exec debain2-1251972161-9po9g ls /var/run/secrets/kubernetes.io/serviceaccount ca.crt namespace token

[cheld]

ok, please try:

kubectl run test --image {container-with-curl, e.g. gcr.io/google_containers/hyperkube-amd64:v1.3.0-beta.1 } sleep 100000

kubectl exec test.... curl -k -u admin:admin https://10.0.0.1:443 kubectl exec test.... curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443

Can you post the result?

Most likely you have inconsistent certificates. Maybe you killed the cluster during boot-up, or something. It should work if you clean up everything:

docker kill $(docker ps -q) docker rm $(docker ps -aq) [reboot] sudo rm -R /var/lib/kubelet sudo rm -R /var/run/kubernetes

[arhided]

$kubectl get pods NAME READY STATUS RESTARTS AGE hello-node-3689398978-2tqvf 1/1 Running 0 29m test-541238630-gby7g 1/1 Running 0 9m

$ kubectl exec test-541238630-gby7g -- curl -k -u admin:admin https://10.0.0.1:443 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0^C

$ kubectl exec test-541238630-gby7g -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0^C

[arhided]

Looks like there is no connectivity between containers.

[arhided]

Got it working, i rebuild kuber-cluster and now it's show's web UI:

http://127.0.0.1:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/pod?namespace=default and http://10.0.0.96/#/workload?namespace=default

Thx.

[bryk]

Oh, perfect! I'm closing the issue. Please reopen if needed.

[sarastinishi]

Still having this issue. Dashboard works fine right after K8s installation, but fails to start upon reboot.

kubectl get pods -n kube-system

kubernetes-dashboard-3543765157-4ftml 0/1 CrashLoopBackOff 1 12s

kubectl logs kubernetes-dashboard-3543765157-4ftml -n kube-system

Using HTTP port: 9090 Creating API server client for https://10.3.0.1:443 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.3.0.1:443/version: dial tcp 10.3.0.1:443: getsockopt: no route to host Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

[cachance7]

For me, the solution was to loosen up overzealous firewall rules preventing the dashboard from accessing the subnet associated with the flannel interface. Because this subnet changed with every docker service restart, it was a few rounds of whack-a-mole before I realized what was going on.

[winse]

open the port 6443 on firewall will be ok.

the follow is the firewall reject log before open the port:

Feb 25 00:04:30 k8s kernel: XFS (dm-32): Mounting V5 Filesystem
Feb 25 00:04:30 k8s kernel: XFS (dm-32): Ending clean mount
Feb 25 00:04:32 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11531 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0 
Feb 25 00:04:33 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11532 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0 
Feb 25 00:04:33 k8s dockerd: time="2017-02-25T00:04:33.935301481+08:00" level=error msg="containerd: deleting container" error="exit status 1: \"container dcb4a44031b96470eaef50eb8ac4ee2b9f958906702d94645c3a45c4852b6335 does not exist\\none or more of the container deletions failed\\n\""
Feb 25 00:04:34 k8s kernel: XFS (dm-32): Unmounting Filesystem

[vhosakot]

I see this issue too with kubernetes 1.5.4 and kubernetes-dashboard image version gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0.

I installed kubeadm referring https://kubernetes.io/docs/getting-started-guides/kubeadm/, and then installed kubernetes-dashboard by doing

kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.0/src/deploy/kubernetes-dashboard.yaml

I see the kubernetes-dashboard in CrashLoopBackOff status and the k8s_kubernetes-dashboard.* container on the worker is in Exited state.

Below are the errors. Has anyone successfully installed kubernetes-dashboard on kubeadm?

# kubectl --namespace=kube-system get all
NAME                                                          READY     STATUS             RESTARTS   AGE
po/calico-policy-controller-mqsmh                             1/1       Running            0          4h
po/canal-etcd-tm2rv                                           1/1       Running            0          4h
po/canal-node-3nv2t                                           3/3       Running            0          4h
po/canal-node-5fckh                                           3/3       Running            1          4h
po/canal-node-6zgq8                                           3/3       Running            0          4h
po/canal-node-rtjl8                                           3/3       Running            0          4h
po/dummy-2088944543-09w8n                                     1/1       Running            0          4h
po/etcd-vhosakot-kolla-kube1.localdomain                      1/1       Running            0          4h
po/kube-apiserver-vhosakot-kolla-kube1.localdomain            1/1       Running            2          4h
po/kube-controller-manager-vhosakot-kolla-kube1.localdomain   1/1       Running            0          4h
po/kube-discovery-1769846148-pftx5                            1/1       Running            0          4h
po/kube-dns-2924299975-9m2cp                                  4/4       Running            0          4h
po/kube-proxy-0ndsb                                           1/1       Running            0          4h
po/kube-proxy-h7qrd                                           1/1       Running            1          4h
po/kube-proxy-k6168                                           1/1       Running            0          4h
po/kube-proxy-lhn0k                                           1/1       Running            0          4h
po/kube-scheduler-vhosakot-kolla-kube1.localdomain            1/1       Running            0          4h
po/kubernetes-dashboard-3203962772-mw26t                      0/1       CrashLoopBackOff   11         41m
NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/canal-etcd             10.96.232.136    <none>        6666/TCP        4h
svc/kube-dns               10.96.0.10       <none>        53/UDP,53/TCP   4h
svc/kubernetes-dashboard   10.100.254.77    <nodes>       80:30085/TCP    41m
NAME                   DESIRED   SUCCESSFUL   AGE
jobs/configure-canal   1         1            4h
NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/kube-discovery         1         1         1            1           4h
deploy/kube-dns               1         1         1            1           4h
deploy/kubernetes-dashboard   1         1         1            0           41m
NAME                                 DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller          1         1         1         4h
rs/dummy-2088944543                  1         1         1         4h
rs/kube-discovery-1769846148         1         1         1         4h
rs/kube-dns-2924299975               1         1         1         4h
rs/kubernetes-dashboard-3203962772   1         1         0         41m

# kubectl --namespace=kube-system describe pod kubernetes-dashboard-3203962772-mw26t
  20m    5s    89    {kubelet vhosakot-kolla-kube2.localdomain}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203962772-mw26t_kube-system(67b0d69b-0b47-11e7-8c97-7a2ed4192438)"

# kubectl --namespace=kube-system logs kubernetes-dashboard-3203962772-mw26t
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

# docker ps -a | grep -i dash
3c33cf43d5e4        gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0   "/dashboard --port=90"   54 seconds ago      Exited (1) 22 seconds ago                       k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4

# docker logs k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

[ironmike-au]

I'm not sure what it is about kubeadm specifically, but I was able to get this working by forcing dashboard to run on master. I did this using nodeSelector in the kubernetes-dashboard.yaml file:

nodeSelector:
  node-role.kubernetes.io/master:

Once I did that and re-added the service, it worked a charm!

[timm088]

@otakumike worked for me after i tainted my master node as per the kubeadm doco kubectl taint nodes --all node-role.kubernetes.io/master-

I think this appears like something to do with with the kubeadm install, but am too noob right now to be able to debug. Any clues about where I should look, happy to start posting anything useful.

[floreks]

By default kubeadm does not allow running applications without specified selector on master node.

First solution is to specify selector as @otakumike did. Second option is to remove taint from the master to allow deploy of applications without this selector as @timm088 did.

- sign at the end of taint removes it.

[Beanking77]

Facing the same issue, too. Install k8s cluster with kubeadm: v1.6.4 and dashboard. solved by using @otakumike way and ran on master node successfully. Cheers!

But are there anyway not run applications on master node? or it would just fine to run applications on master node?

[floreks]

Kubeadm by default taints master node so all applications without proper toleration set can not be started on master node.

kubectl taint nodes --all node-role.kubernetes.io/master- This removed the default taint and allows to run applications on master. Dashboard does not have to be run on master node. It might be any node in the cluster. Usually you shouldn't run normal applications on master unless you need to.

[Beanking77]

@floreks thanks for your quick reply. But recently I can only run dashboard successfully on master node, same error logs as @vhosakot when dashboard is on another nodes. Is this would be a issue?

[floreks]

If there is a connection timeout/refused error then usually it means that there is a problem with networking/dns in your cluster. For in-cluster config dashboard uses service proxy to access apiserver and for that to work service discovery and dns have to work correctly.

The easiest way to check that is to create busybox container and do an nslookup like described here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

[Beanking77]

I got some error logs when verify DNS setting, and been solved by add (OS: Centos 7.3)

iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT

it looks not good to me,

$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'

and still got error when I try to run dashboard on another node.

docker logs k8s_kubernetes-dashboard_kubernetes-dashboard-2039414953-qdbwx_kube-system_812c20bf-572a-11e7-b95d-b8aeed766268_2
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: getsockopt: no route to host
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

Should I need to delete DNS pod and auto-create it?

[floreks]

Definitely you have some kind of networking issue in your cluster and dashboard can't access apiserver. Try checking existing/closed issues on kubernetes repository. I think simple pod deletion won't resolve this. It might be related to system/firewall.

[Beanking77]

@floreks Thanks you so much for your reply and now dashboard working on minion perfectly.

OS: CentOS 7.3 stop the firewall

$ systemctl stop firewalld
$ systemctl disable firewalld

and make sure /usr/lib/sysctl.d/00-system.conf config are

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

and iptable rules are

iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT

[floreks]

No problem. Great to hear that it works!

[ronanquillevere]

Hello,

I am having the same issue, I have created a cluster with one master (server-1) and one minion(server-2) running on ubuntu and using flannel for networking and using kubeadm. And I am trying to run the dashboard from the minion server-2. I am very new to kubernetes and not an expert on linux networking setup so any help would be appreciated.

kubectl and kubeadm : 1.6.6 Docker: 17.03.1-ce

This is the command I followed to install my cluster 1) Installing master kubeadm init --pod-network-cidr=10.244.0.0/16 2) Installing flannel kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml Setting-up RBAC kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml 3) Joining from minion kubeadm join --token <token> <master-ip>:<master-port> 4) Installing Dashboard kubectl create -f https://git.io/kube-dashboard

Now here are all the pods I see :

NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
kube-system   etcd-server-1                           1/1       Running            0          15h
kube-system   kube-apiserver-server-1                 1/1       Running            0          15h
kube-system   kube-controller-manager-server-1        1/1       Running            0          15h
kube-system   kube-dns-692378583-5zj21                3/3       Running            0          15h
kube-system   kube-flannel-ds-df0f8                   2/2       Running            4          15h
kube-system   kube-flannel-ds-dzkqk                   2/2       Running            0          15h
kube-system   kube-proxy-gwbjj                        1/1       Running            0          15h
kube-system   kube-proxy-sx8qb                        1/1       Running            0          15h
kube-system   kube-scheduler-server-1                 1/1       Running            0          15h
kube-system   kubernetes-dashboard-2039414953-8frcf   0/1       CrashLoopBackOff   179        15h

Here are the services I see

NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes             10.96.0.1        <none>        443/TCP         15h
kube-system   kube-dns               10.96.0.10       <none>        53/UDP,53/TCP   15h
kube-system   kubernetes-dashboard   10.103.141.154   <none>        80/TCP          15h

Dasbard is in error with

Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

Dashboard is running on minion (server-2), here is the desciption of the pod (just removed the real ip)

Name:       kubernetes-dashboard-2039414953-8frcf
Namespace:  kube-system
Node:       server-2/xxx.xxx.xxx.xxx
Start Time: Tue, 27 Jun 2017 16:44:08 +0000
Labels:     k8s-app=kubernetes-dashboard
        pod-template-hash=2039414953
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-2039414953","uid":"d7031736-5b57-11e7-8...
Status:     Running
IP:     10.244.1.19
Controllers:    ReplicaSet/kubernetes-dashboard-2039414953
Containers:
  kubernetes-dashboard:
    Container ID:   docker://1776d490980e860f97ab2ad94783e10c30166a5e6c53f72bfe0d6161e9b10863
    Image:      gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1
    Image ID:       docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:b537ce8988510607e95b8d40ac9824523b1f9029e6f9f90e9fccc663c355cf5d
    Port:       9090/TCP
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 28 Jun 2017 08:14:23 +0000
      Finished:     Wed, 28 Jun 2017 08:14:53 +0000
    Ready:      False
    Restart Count:  180
    Liveness:       http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-rnvnc (ro)
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  kubernetes-dashboard-token-rnvnc:
    Type:   Secret (a volume populated by a Secret)
    SecretName: kubernetes-dashboard-token-rnvnc
    Optional:   false
QoS Class:  BestEffort
Node-Selectors: <none>
Tolerations:    node-role.kubernetes.io/master=:NoSchedule
        node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath               Type        Reason      Message
  --------- --------    -----   ----            -------------               --------    ------      -------
  15h       2m      181 kubelet, server-2   spec.containers{kubernetes-dashboard}   Normal      Pulled      Container image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1" already present on machine
  15h       2m      172 kubelet, server-2   spec.containers{kubernetes-dashboard}   Normal      Created     (events with common reason combined)
  15h       2m      172 kubelet, server-2   spec.containers{kubernetes-dashboard}   Normal      Started     (events with common reason combined)
  15h       11s     3893    kubelet, server-2   spec.containers{kubernetes-dashboard}   Warning     BackOff     Back-off restarting failed container
  15h       11s     3863    kubelet, server-2                       Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-2039414953-8frcf_kube-system(d708bf94-5b57-11e7-89e4-fa163ebaf0b4)"

Here is iptables from server 1

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-ISOLATION  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             10.103.141.154       /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable

here are iptables from server-2

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-ISOLATION  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             10.103.141.154       /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable

I have been looking at the following troubleshooting guide but I could not find a solution (maybe I missed something) : https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

---

The starting error I got was when trying to run kubectl proxy and open following url from my workstation I get

http://127.0.0.1:8001/ui/

Get redirect (307) to

http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "no endpoints available for service \"kubernetes-dashboard\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

[ironmike-au]

@ronanquillevere Did you see my update on this from 16 days prior, about running dashboard on the master? I think that should solve your problem.

[ronanquillevere]

@otakumike I am trying to do that right now to move on but still I would like to run it on the minion and understand WHY i is not working right now :)

The point is I have other issues when trying to run other kind of services which are maybe also related, I think there is a networking problem that I do not understand, but again I am a newbee.

[ironmike-au]

@ronanquillevere Me too! :) If you find a solution to that please do post it here.

[floreks]

This is not a dashboard issue. First check you cluster networking and dns: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

There is a simple busybox container that allows you to do basic networking/dns check. Run the container and execute kubectl exec -ti busybox -- nslookup kubernetes.default. If this fails to resolve then you have a cluster issue and any application that would need to connect to other app in the cluster through services will fail.

For this kind of issues create one in core repository. They are more experienced in this area.

[Beanking77]

@otakumike @ronanquillevere I faced the same issue and after I followed @floreks suggest to check dns/firewall and fixed it by some steps I post 5 days ago, and the dashboard can works on minion now. Maybe you could try the same way in ubuntu.

pod list: (test-kube01 - master)

NAME                                       READY     STATUS    RESTARTS   AGE       IP            NODE
default-http-backend-2198840601-nhj46      1/1       Running   0          4d        10.244.2.7    test-kube03
etcd-test-kube01                           1/1       Running   0          5d        10.1.30.14    test-kube01
heapster-2315332064-x7zn5                  1/1       Running   0          5d        10.244.2.3    test-kube03
kube-apiserver-test-kube01                 1/1       Running   0          5d        10.1.30.14    test-kube01
kube-controller-manager-test-kube01        1/1       Running   0          5d        10.1.30.14    test-kube01
kube-dns-3913472980-hhkkx                  3/3       Running   0          5d        10.244.0.25   test-kube01
kube-flannel-ds-9nkxq                      2/2       Running   0          5d        10.1.30.15    test-kube02
kube-flannel-ds-jf3bk                      2/2       Running   0          5d        10.1.30.16    test-kube03
kube-flannel-ds-k4ctc                      2/2       Running   0          5d        10.1.30.14    test-kube01
kube-proxy-l2cjw                           1/1       Running   0          5d        10.1.30.14    test-kube01
kube-proxy-lg6b4                           1/1       Running   0          5d        10.1.30.15    test-kube02
kube-proxy-mx057                           1/1       Running   0          5d        10.1.30.16    test-kube03
kube-scheduler-test-kube01                 1/1       Running   0          5d        10.1.30.14    test-kube01
kubernetes-dashboard-2039414953-65392      1/1       Running   0          5d        10.244.2.2    test-kube03
monitoring-grafana-3975459543-c9pl1        1/1       Running   0          5d        10.244.3.3    test-kube02
monitoring-influxdb-3480804314-lnrr3       1/1       Running   0          5d        10.244.3.4    test-kube02
nginx-ingress-controller-429549519-q8jvl   1/1       Running   0          4d        10.1.30.15    test-kube02
weave-scope-agent-6chk2                    1/1       Running   0          4d        10.1.30.16    test-kube03
weave-scope-agent-dvgz7                    1/1       Running   0          4d        10.1.30.15    test-kube02
weave-scope-agent-zqpfb                    1/1       Running   0          4d        10.1.30.14    test-kube01
weave-scope-app-879505699-4lwx9            1/1       Running   0          4d        10.244.2.8    test-kube03

dns test with kubectl exec -ti busybox -- nslookup kubernetes.default

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

[ronanquillevere]

@Beanking77

added the iptables rules on master (server-1). But on my servers, no firewalld is running (see list below of enabled services) and no such file : /usr/lib/sysctl.d/00-system.conf

sudo ufw status
Status: inactive

It is not working better but I am probably missing something (I am a newbie on linux networking config).

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10250
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:9898
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:6443
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8472
KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere    
...

ubuntu@server-1:~$ systemctl list-unit-files | grep enabled
acpid.path                             enabled  
accounts-daemon.service                enabled  
apparmor.service                       enabled  
atd.service                            enabled  
autovt@.service                        enabled  
cloud-config.service                   enabled  
cloud-final.service                    enabled  
cloud-init-local.service               enabled  
cloud-init.service                     enabled  
console-setup.service                  enabled  
cron.service                           enabled  
dbus-org.freedesktop.resolve1.service  enabled  
docker.service                         enabled  
ebtables.service                       enabled  
friendly-recovery.service              enabled  
getty@.service                         enabled  
iscsi.service                          enabled  
iscsid.service                         enabled  
keyboard-setup.service                 enabled  
kubelet.service                        enabled  
lvm2-monitor.service                   enabled  
lxcfs.service                          enabled  
lxd-containers.service                 enabled  
networking.service                     enabled  
ondemand.service                       enabled  
open-iscsi.service                     enabled  
open-vm-tools.service                  enabled  
pollinate.service                      enabled  
resolvconf.service                     enabled  
rsyslog.service                        enabled  
setvtrgb.service                       enabled  
snapd.autoimport.service               enabled  
snapd.service                          enabled  
snapd.system-shutdown.service          enabled  
ssh.service                            enabled  
sshd.service                           enabled  
syslog.service                         enabled  
systemd-resolved.service               enabled  
systemd-timesyncd.service              enabled  
ufw.service                            enabled  
unattended-upgrades.service            enabled  
ureadahead.service                     enabled  
acpid.socket                           enabled  
apport-forward.socket                  enabled  
dm-event.socket                        enabled  
docker.socket                          enabled  
lvm2-lvmetad.socket                    enabled  
lvm2-lvmpolld.socket                   enabled  
lxd.socket                             enabled  
snapd.socket                           enabled  
uuidd.socket                           enabled  
remote-fs.target                       enabled  
apt-daily.timer                        enabled  
snapd.refresh.timer                    enabled  

[Beanking77]

@ronanquillevere According to https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ at limitation section, only RHEL/CentOS 7 user might to edit that file (I'm CentOS 7.3).

Have you test your network, like test dns with kubectl exec -ti busybox -- nslookup kubernetes.default. Make sure network should work properly and the dashboard should work on minion.

[ronanquillevere]

@Beanking77 @floreks I am looking at that right now, try to fix my DNS :)

kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'

But everything looks fine

kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local local
options ndots:5

 kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                       READY     STATUS    RESTARTS   AGE
kube-dns-692378583-5zj21   3/3       Running   0          17h

I got not error/warnings in the logs of kubedns and dnsmasq. I do not have the healthz container but I believe this is not an issue.

And my DNS service is up an exposing endpoins

ubuntu@server-1:~$ kubectl get svc --namespace=kube-system
NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kube-dns               10.96.0.10      <none>        53/UDP,53/TCP   17h
kubernetes-dashboard   10.97.135.242   <none>        80/TCP          1h
ubuntu@server-1:~$ kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                     AGE
kube-dns   10.244.0.4:53,10.244.0.4:53   17h

Created a new issue https://github.com/kubernetes/dns/issues/118

[Beanking77]

@ronanquillevere In my experienced in CentOS, it might be firewall or ip table rule issue. But I'm not quite sure if it happen in ubuntu, too. It might be related to system/firewall.

[spnzig]

I have a similar issue with kubernetes dashboard. I have used kubeadm for k8s install and then installed dashboard using kubectl apply -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml. But from a long time ( more than 40 minutes) the state is still in ContainerCreating. I don't seem to understand the error. Can someone help me with this please.

When i try to check the logs with kubectl logs --namespace=kube-system kubernetes-dashboard-xxxxx i get Error from server (Bad Request): container "kubernetes-dashboard" in pod "kubernetes-dasshboard-xxx" is waiting to start: ContainerCreating

@floreks i did install busybox as per the link you mentioned. But even there the pod busybox remains in pending state. Could you please suggest on how I can proceed? With this i get pod busybox does not have a host assigned

[floreks]

Did you taint the node to allow application deployment on master? kubectl taint nodes --all node-role.kubernetes.io/master-

[spnzig]

@floreks it gives error: at least one taint update is required

[floreks]

The instead of --all specify node name you want to taint.

[spnzig]

@floreks I get the same error with kubectl taint nodes "master_node_name" node-role.kubernetes.io/master. I am a newbie to K8S. So could you please elaborate

[floreks]

Did you forgot - sign at the end of command?

[spnzig]

Oh i didn't notice that. Now i get node "master-node-name" untainted. Now after running the kube-proxy command i get 503 service unavailable error

[floreks]

This is not a dashboard issue but cluster. As you have used kubeadm please try to ask for help on https://github.com/kubernetes/kubeadm or https://github.com/kubernetes/kubernetes. We are dealing here only with dashboard specific problems.