Closed Rahul91 closed 8 years ago
Is the Dashboard UI running? Can you show me logs of the dashboard pod?
I got the same error.
Starting HTTP server on port 9090 Creating API server client for https://10.0.0.1:443 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.0.0.1:443/version: dial tcp 10.0.0.1:443: i/o timeout
Name: kubernetes-dashboard
Namespace: kube-system
Labels: k8s-app=kubernetes-dashboard,kubernetes.io/cluster-service=true Selector: k8s-app=kubernetes-dashboard
Type:
ClusterIP IP: 10.0.0.55
Port:
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1
@bryk Gist: https://gist.github.com/Rahul91/f443e58dd730e0571bcea6409adb5761 I am getting this error in http://master_ip/ui
Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'
I am running my master on a server with publicly accessible IP and minion on my local machine running in a local network. Is that the reason I am getting this error?
@Rahul91 As your master is located on external server try to provide apiserver-host
parameter to the dashboard. It is commented inside the yaml file. Without providing it, dashboard tries to discover master node locally.
@arhided how are you running your cluster? Locally? AWS/GCE?
Kube is running locally.
~$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1
I think Kube service is there and kube-dash autodiscovery also locationg it properly.
@arhided
Is it local-up-cluster
script or kube-up
?
autodiscovery is looking for master on port local-up-cluster
does not offer https
endpoint as far as i know. It is only exposed using http
protocol and443
which is https
.
To make it work with dashboard locally you may want to use hyperkube script that we have on our repository in build
folder. Other issue may be that docker daemon is not configured properly and dashboard container can not access kubernetes master.
@bryk any other ideas?
hack/local-up-cluster.sh Go version: go version go1.6.2 linux/amd64 +++ [0622 17:56:05] Building the toolchain targets: k8s.io/kubernetes/hack/cmd/teststale +++ [0622 17:56:05] Building go targets for linux/amd64: cmd/kubectl cmd/hyperkube +++ [0622 17:56:07] Placing binaries curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused API SERVER port is free, proceeding... Detected host and ready to start services. Doing some housekeeping first... Using GO_OUT /home/user/kubernetes/_output/local/bin/linux/amd64 Starting services now! Starting etcd etcd -addr 127.0.0.1:4001 -data-dir /tmp/tmp.hXTnA9YpPY --bind-addr 127.0.0.1:4001 >/dev/null 2>/dev/null Waiting for etcd to come up. +++ [0622 17:56:08] On try 2, etcd: : {"action":"set","node":{"key":"/_test","value":"","modifiedIndex":3,"createdIndex":3}} Waiting for apiserver to come up +++ [0622 17:56:09] On try 2, apiserver: : { "kind": "PodList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/pods", "resourceVersion": "10" }, "items": [] } Local Kubernetes cluster is running. Press Ctrl-C to shut it down.
Logs: /tmp/kube-apiserver.log /tmp/kube-controller-manager.log /tmp/kube-proxy.log /tmp/kube-scheduler.log /tmp/kubelet.log
To start using your cluster, open up another terminal/tab and run:
cluster/kubectl.sh config set-cluster local --server=http://127.0.0.1:8080 --insecure-skip-tls-verify=true cluster/kubectl.sh config set-context local --cluster=local cluster/kubectl.sh config use-context local cluster/kubectl.sh
Http it there
{ "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1alpha1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/extensions", "/apis/extensions/v1beta1", "/apis/policy", "/apis/policy/v1alpha1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/healthz", "/healthz/ping", "/logs/", "/metrics", "/swaggerapi/", "/ui/", "/version" ] }
An error occurred during a connection to 127.0.0.1:8080. SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long)
Hypercube.sh - don`t change much, pretty much nothing.
gcr.io/google_containers/heapster:v1.0.2 "/heapster -port 8082"
http://127.0.0.1:8082/ 404 page not found
@floreks, @bryk I ran kubernetes-dashboard.yaml on my master server, with --apiserver-host commented and even ran by uncommenting and gave public IP of my server. Both of above didn't work.
On my minion server(running in a private network)
$ps -ef | grep kube
root 17834 1 1 14:24 ? 00:00:06 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=0.0.0.0 --api_servers=http://163.172.128.148:8080 --logtostderr=true
root 17851 1 0 14:24 ? 00:00:01 /opt/bin/kube-proxy --master=163.172.128.148:8080 --logtostderr=true
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aa4d35d4898f gcr.io/google_containers/kubernetes-dashboard-amd64:v1.1.0 "/dashboard --port=90" About a minute ago Up About a minute k8s_kubernetes-dashboard.9ae56f1c_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_d5712460
2dd21a03d9d2 gcr.io/google_containers/pause:0.8.0 "/pause" About a minute ago Up About a minute k8s_POD.3e8ce8bb_kubernetes-dashboard-mc01o_kube-system_280da0d1-3956-11e6-9ed6-0007cb03d000_b9769263
$ docker logs aa4d35d4898f
Starting HTTP server on port 9090
Creating API server client for http://163.172.128.148:8080
Successful initial request to the apiserver, version: v1.2.4
Creating in-cluster Heapster client
On my master node(running on a server having public accessible IP)
# ps -ef | grep kube
root 16620 1 0 07:26 ? 00:00:01 /opt/bin/kube-scheduler --master=127.0.0.1:8080 --logtostderr=true --v=0
root 16621 1 0 07:26 ? 00:00:19 /opt/bin/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080 --etcd_servers=http://10.1.72.219:2379 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --secure-port=0 --service-node-port-range=30000-32767 --service_account_lookup=false
root 16623 1 0 07:26 ? 00:00:17 /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --v=0
#kubectl get rc --namespace=kube-system
NAME DESIRED CURRENT AGE
kubernetes-dashboard 1 1 19h
# kubectl get po --namespace=kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-mc01o 1/1 Running 0 17h
#kubectl logs kubernetes-dashboard-mc01o
Error from server: pods "kubernetes-dashboard-mc01o" not found
root@onehop-for-docker-testing:~# kubectl logs kubernetes-dashboard-mc01o --namespace=kube-system
Error from server: Get https://0.0.0.0:10250/containerLogs/kube-system/kubernetes-dashboard-mc01o/kubernetes-dashboard: dial tcp 0.0.0.0:10250: connection refused
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/resetMetrics",
"/swagger-ui/",
"/swaggerapi/",
"/ui/",
"/version"
]
}
http://master_ip:8080/api/v1/namespaces/kube-system/services/kubernetes-dashboard/
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "kubernetes-dashboard",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
"uid": "b31b3fbf-3943-11e6-9ed6-0007cb03d000",
"resourceVersion": "210445",
"creationTimestamp": "2016-06-23T13:09:21Z",
"labels": {
"app": "kubernetes-dashboard"
}
},
"spec": {
"ports": [
{
"protocol": "TCP",
"port": 80,
"targetPort": 9090,
"nodePort": 31260
}
],
"selector": {
"app": "kubernetes-dashboard"
},
"clusterIP": "192.168.3.63",
"type": "NodePort",
"sessionAffinity": "None"
},
"status": {
"loadBalancer": {}
}
}
http://master_ip:8080/api/v1/**proxy**/namespaces/kube-system/services/kubernetes-dashboard/
Error: 'dial tcp 10.100.22.2:9090: i/o timeout'
Trying to reach: 'http://10.100.22.2:9090/'
I am new to kubernetes, please help me resolve this issue.
$ docker logs aa4d35d4898f Starting HTTP server on port 9090 Creating API server client for http://163.172.128.148:8080 Successful initial request to the apiserver, version: v1.2.4 Creating in-cluster Heapster client
This means that the UI successfully connected to the apiserver. From now on it should just work. I guess this is something wrong with your cluster and proxy settings. Can you change dashboard service from "NodePort" to "LoadBalancer". This would give it an external ip address that you can access.
@arhided
which docker version are you using? which kubernetes version?
could you please paste the output of: kubectl run debain2 --image debian sleep 1000000 kubectl exec debain2-xxxx ls /var/run/secrets/kubernetes.io/serviceaccount
Hi
$ docker version Client: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:16:54 UTC 2015 OS/Arch: linux/amd64
Server: Version: 1.9.1 API version: 1.21 Go version: go1.4.2 Git commit: a34a1d5 Built: Fri Nov 20 13:16:54 UTC 2015 OS/Arch: linux/amd64
http://127.0.0.1:8080/version { "major": "1", "minor": "3+", "gitVersion": "v1.3.0-alpha.4.893+2976e892a45fb2-dirty", "gitCommit": "2976e892a45fb2fdaf458af1c90040c095feb0d1", "gitTreeState": "dirty", "buildDate": "2016-06-20T14:02:32Z", "goVersion": "go1.6.2", "compiler": "gc", "platform": "linux/amd64" }
$ kubectl run debain2 --image debian sleep 1000000 deployment "debain2" created
$ kubectl exec debain2-1251972161-9po9g ls /var/run/secrets/kubernetes.io/serviceaccount ca.crt namespace token
ok, please try:
kubectl run test --image {container-with-curl, e.g. gcr.io/google_containers/hyperkube-amd64:v1.3.0-beta.1 } sleep 100000
kubectl exec test.... curl -k -u admin:admin https://10.0.0.1:443 kubectl exec test.... curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443
Can you post the result?
Most likely you have inconsistent certificates. Maybe you killed the cluster during boot-up, or something. It should work if you clean up everything:
docker kill $(docker ps -q) docker rm $(docker ps -aq) [reboot] sudo rm -R /var/lib/kubelet sudo rm -R /var/run/kubernetes
$kubectl get pods NAME READY STATUS RESTARTS AGE hello-node-3689398978-2tqvf 1/1 Running 0 29m test-541238630-gby7g 1/1 Running 0 9m
$ kubectl exec test-541238630-gby7g -- curl -k -u admin:admin https://10.0.0.1:443 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0^C
$ kubectl exec test-541238630-gby7g -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -u admin:admin https://10.0.0.1:443 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0^C
Looks like there is no connectivity between containers.
Got it working, i rebuild kuber-cluster and now it's show's web UI:
http://127.0.0.1:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/pod?namespace=default and http://10.0.0.96/#/workload?namespace=default
Thx.
Oh, perfect! I'm closing the issue. Please reopen if needed.
Still having this issue. Dashboard works fine right after K8s installation, but fails to start upon reboot.
kubectl get pods -n kube-system
kubernetes-dashboard-3543765157-4ftml 0/1 CrashLoopBackOff 1 12s
kubectl logs kubernetes-dashboard-3543765157-4ftml -n kube-system
Using HTTP port: 9090 Creating API server client for https://10.3.0.1:443 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.3.0.1:443/version: dial tcp 10.3.0.1:443: getsockopt: no route to host Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
For me, the solution was to loosen up overzealous firewall rules preventing the dashboard from accessing the subnet associated with the flannel interface. Because this subnet changed with every docker service restart, it was a few rounds of whack-a-mole before I realized what was going on.
open the port 6443 on firewall will be ok.
the follow is the firewall reject log before open the port:
Feb 25 00:04:30 k8s kernel: XFS (dm-32): Mounting V5 Filesystem
Feb 25 00:04:30 k8s kernel: XFS (dm-32): Ending clean mount
Feb 25 00:04:32 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11531 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0
Feb 25 00:04:33 k8s kernel: FINAL_REJECT: IN=docker0 OUT= PHYSIN=veth2fd9745 MAC=02:42:cf:c5:2c:da:02:42:0a:01:49:03:08:00 SRC=10.1.73.3 DST=192.168.1.112 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11532 DF PROTO=TCP SPT=38734 DPT=6443 WINDOW=28640 RES=0x00 SYN URGP=0
Feb 25 00:04:33 k8s dockerd: time="2017-02-25T00:04:33.935301481+08:00" level=error msg="containerd: deleting container" error="exit status 1: \"container dcb4a44031b96470eaef50eb8ac4ee2b9f958906702d94645c3a45c4852b6335 does not exist\\none or more of the container deletions failed\\n\""
Feb 25 00:04:34 k8s kernel: XFS (dm-32): Unmounting Filesystem
I see this issue too with kubernetes 1.5.4
and kubernetes-dashboard image version gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0
.
I installed kubeadm
referring https://kubernetes.io/docs/getting-started-guides/kubeadm/, and then installed kubernetes-dashboard by doing
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.0/src/deploy/kubernetes-dashboard.yaml
I see the kubernetes-dashboard in CrashLoopBackOff
status and the k8s_kubernetes-dashboard.*
container on the worker is in Exited
state.
Below are the errors. Has anyone successfully installed kubernetes-dashboard on kubeadm
?
# kubectl --namespace=kube-system get all
NAME READY STATUS RESTARTS AGE
po/calico-policy-controller-mqsmh 1/1 Running 0 4h
po/canal-etcd-tm2rv 1/1 Running 0 4h
po/canal-node-3nv2t 3/3 Running 0 4h
po/canal-node-5fckh 3/3 Running 1 4h
po/canal-node-6zgq8 3/3 Running 0 4h
po/canal-node-rtjl8 3/3 Running 0 4h
po/dummy-2088944543-09w8n 1/1 Running 0 4h
po/etcd-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kube-apiserver-vhosakot-kolla-kube1.localdomain 1/1 Running 2 4h
po/kube-controller-manager-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kube-discovery-1769846148-pftx5 1/1 Running 0 4h
po/kube-dns-2924299975-9m2cp 4/4 Running 0 4h
po/kube-proxy-0ndsb 1/1 Running 0 4h
po/kube-proxy-h7qrd 1/1 Running 1 4h
po/kube-proxy-k6168 1/1 Running 0 4h
po/kube-proxy-lhn0k 1/1 Running 0 4h
po/kube-scheduler-vhosakot-kolla-kube1.localdomain 1/1 Running 0 4h
po/kubernetes-dashboard-3203962772-mw26t 0/1 CrashLoopBackOff 11 41m
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/canal-etcd 10.96.232.136 <none> 6666/TCP 4h
svc/kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 4h
svc/kubernetes-dashboard 10.100.254.77 <nodes> 80:30085/TCP 41m
NAME DESIRED SUCCESSFUL AGE
jobs/configure-canal 1 1 4h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/kube-discovery 1 1 1 1 4h
deploy/kube-dns 1 1 1 1 4h
deploy/kubernetes-dashboard 1 1 1 0 41m
NAME DESIRED CURRENT READY AGE
rs/calico-policy-controller 1 1 1 4h
rs/dummy-2088944543 1 1 1 4h
rs/kube-discovery-1769846148 1 1 1 4h
rs/kube-dns-2924299975 1 1 1 4h
rs/kubernetes-dashboard-3203962772 1 1 0 41m
# kubectl --namespace=kube-system describe pod kubernetes-dashboard-3203962772-mw26t
20m 5s 89 {kubelet vhosakot-kolla-kube2.localdomain} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203962772-mw26t_kube-system(67b0d69b-0b47-11e7-8c97-7a2ed4192438)"
# kubectl --namespace=kube-system logs kubernetes-dashboard-3203962772-mw26t
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
# docker ps -a | grep -i dash
3c33cf43d5e4 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0 "/dashboard --port=90" 54 seconds ago Exited (1) 22 seconds ago k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
# docker logs k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
I'm not sure what it is about kubeadm specifically, but I was able to get this working by forcing dashboard to run on master. I did this using nodeSelector in the kubernetes-dashboard.yaml file:
nodeSelector:
node-role.kubernetes.io/master:
Once I did that and re-added the service, it worked a charm!
@otakumike worked for me after i tainted my master node as per the kubeadm doco
kubectl taint nodes --all node-role.kubernetes.io/master-
I think this appears like something to do with with the kubeadm install, but am too noob right now to be able to debug. Any clues about where I should look, happy to start posting anything useful.
By default kubeadm does not allow running applications without specified selector on master node.
First solution is to specify selector as @otakumike did. Second option is to remove taint from the master to allow deploy of applications without this selector as @timm088 did.
-
sign at the end of taint removes it.
Facing the same issue, too. Install k8s cluster with kubeadm: v1.6.4 and dashboard. solved by using @otakumike way and ran on master node successfully. Cheers!
But are there anyway not run applications on master node? or it would just fine to run applications on master node?
Kubeadm by default taints master node so all applications without proper toleration set can not be started on master node.
kubectl taint nodes --all node-role.kubernetes.io/master-
This removed the default taint and allows to run applications on master. Dashboard does not have to be run on master node. It might be any node in the cluster. Usually you shouldn't run normal applications on master unless you need to.
@floreks thanks for your quick reply. But recently I can only run dashboard successfully on master node, same error logs as @vhosakot when dashboard is on another nodes. Is this would be a issue?
If there is a connection timeout/refused error then usually it means that there is a problem with networking/dns in your cluster. For in-cluster config dashboard uses service proxy to access apiserver and for that to work service discovery and dns have to work correctly.
The easiest way to check that is to create busybox container and do an nslookup
like described here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
I got some error logs when verify DNS setting, and been solved by add (OS: Centos 7.3)
iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
it looks not good to me,
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
and still got error when I try to run dashboard on another node.
docker logs k8s_kubernetes-dashboard_kubernetes-dashboard-2039414953-qdbwx_kube-system_812c20bf-572a-11e7-b95d-b8aeed766268_2
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: getsockopt: no route to host
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
Should I need to delete DNS pod and auto-create it?
Definitely you have some kind of networking issue in your cluster and dashboard can't access apiserver. Try checking existing/closed issues on kubernetes repository. I think simple pod deletion won't resolve this. It might be related to system/firewall.
@floreks Thanks you so much for your reply and now dashboard working on minion perfectly.
OS: CentOS 7.3 stop the firewall
$ systemctl stop firewalld
$ systemctl disable firewalld
and make sure /usr/lib/sysctl.d/00-system.conf config are
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
and iptable rules are
iptables -I INPUT -p tcp -m tcp --dport 8472 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 9898 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
No problem. Great to hear that it works!
Hello,
I am having the same issue, I have created a cluster with one master (server-1
) and one minion(server-2
) running on ubuntu and using flannel for networking and using kubeadm
. And I am trying to run the dashboard from the minion server-2
. I am very new to kubernetes and not an expert on linux networking setup so any help would be appreciated.
kubectl
and kubeadm
: 1.6.6
Docker
: 17.03.1-ce
This is the command I followed to install my cluster
1) Installing master
kubeadm init --pod-network-cidr=10.244.0.0/16
2) Installing flannel
kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
Setting-up RBAC
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
3) Joining from minion
kubeadm join --token <token> <master-ip>:<master-port>
4) Installing Dashboard
kubectl create -f https://git.io/kube-dashboard
Now here are all the pods I see :
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-server-1 1/1 Running 0 15h
kube-system kube-apiserver-server-1 1/1 Running 0 15h
kube-system kube-controller-manager-server-1 1/1 Running 0 15h
kube-system kube-dns-692378583-5zj21 3/3 Running 0 15h
kube-system kube-flannel-ds-df0f8 2/2 Running 4 15h
kube-system kube-flannel-ds-dzkqk 2/2 Running 0 15h
kube-system kube-proxy-gwbjj 1/1 Running 0 15h
kube-system kube-proxy-sx8qb 1/1 Running 0 15h
kube-system kube-scheduler-server-1 1/1 Running 0 15h
kube-system kubernetes-dashboard-2039414953-8frcf 0/1 CrashLoopBackOff 179 15h
Here are the services I see
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.96.0.1 <none> 443/TCP 15h
kube-system kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 15h
kube-system kubernetes-dashboard 10.103.141.154 <none> 80/TCP 15h
Dasbard is in error with
Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
Dashboard is running on minion (server-2), here is the desciption of the pod (just removed the real ip)
Name: kubernetes-dashboard-2039414953-8frcf
Namespace: kube-system
Node: server-2/xxx.xxx.xxx.xxx
Start Time: Tue, 27 Jun 2017 16:44:08 +0000
Labels: k8s-app=kubernetes-dashboard
pod-template-hash=2039414953
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kubernetes-dashboard-2039414953","uid":"d7031736-5b57-11e7-8...
Status: Running
IP: 10.244.1.19
Controllers: ReplicaSet/kubernetes-dashboard-2039414953
Containers:
kubernetes-dashboard:
Container ID: docker://1776d490980e860f97ab2ad94783e10c30166a5e6c53f72bfe0d6161e9b10863
Image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1
Image ID: docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:b537ce8988510607e95b8d40ac9824523b1f9029e6f9f90e9fccc663c355cf5d
Port: 9090/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 28 Jun 2017 08:14:23 +0000
Finished: Wed, 28 Jun 2017 08:14:53 +0000
Ready: False
Restart Count: 180
Liveness: http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kubernetes-dashboard-token-rnvnc (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
kubernetes-dashboard-token-rnvnc:
Type: Secret (a volume populated by a Secret)
SecretName: kubernetes-dashboard-token-rnvnc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master=:NoSchedule
node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15h 2m 181 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Pulled Container image "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1" already present on machine
15h 2m 172 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Created (events with common reason combined)
15h 2m 172 kubelet, server-2 spec.containers{kubernetes-dashboard} Normal Started (events with common reason combined)
15h 11s 3893 kubelet, server-2 spec.containers{kubernetes-dashboard} Warning BackOff Back-off restarting failed container
15h 11s 3863 kubelet, server-2 Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-2039414953-8frcf_kube-system(d708bf94-5b57-11e7-89e4-fa163ebaf0b4)"
Here is iptables from server 1
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp -- anywhere 10.103.141.154 /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable
here are iptables from server-2
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-ISOLATION all -- anywhere anywhere
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp -- anywhere 10.103.141.154 /* kube-system/kubernetes-dashboard: has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable
I have been looking at the following troubleshooting guide but I could not find a solution (maybe I missed something) : https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md
The starting error I got was when trying to run kubectl proxy
and open following url from my workstation I get
Get redirect (307) to
http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "no endpoints available for service \"kubernetes-dashboard\"",
"reason": "ServiceUnavailable",
"code": 503
}
@ronanquillevere Did you see my update on this from 16 days prior, about running dashboard on the master? I think that should solve your problem.
@otakumike I am trying to do that right now to move on but still I would like to run it on the minion and understand WHY i is not working right now :)
The point is I have other issues when trying to run other kind of services which are maybe also related, I think there is a networking problem that I do not understand, but again I am a newbee.
@ronanquillevere Me too! :) If you find a solution to that please do post it here.
This is not a dashboard issue. First check you cluster networking and dns: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
There is a simple busybox container that allows you to do basic networking/dns check. Run the container and execute kubectl exec -ti busybox -- nslookup kubernetes.default
. If this fails to resolve then you have a cluster issue and any application that would need to connect to other app in the cluster through services will fail.
For this kind of issues create one in core repository. They are more experienced in this area.
@otakumike @ronanquillevere I faced the same issue and after I followed @floreks suggest to check dns/firewall and fixed it by some steps I post 5 days ago, and the dashboard can works on minion now. Maybe you could try the same way in ubuntu.
pod list: (test-kube01 - master)
NAME READY STATUS RESTARTS AGE IP NODE default-http-backend-2198840601-nhj46 1/1 Running 0 4d 10.244.2.7 test-kube03 etcd-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 heapster-2315332064-x7zn5 1/1 Running 0 5d 10.244.2.3 test-kube03 kube-apiserver-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-controller-manager-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-dns-3913472980-hhkkx 3/3 Running 0 5d 10.244.0.25 test-kube01 kube-flannel-ds-9nkxq 2/2 Running 0 5d 10.1.30.15 test-kube02 kube-flannel-ds-jf3bk 2/2 Running 0 5d 10.1.30.16 test-kube03 kube-flannel-ds-k4ctc 2/2 Running 0 5d 10.1.30.14 test-kube01 kube-proxy-l2cjw 1/1 Running 0 5d 10.1.30.14 test-kube01 kube-proxy-lg6b4 1/1 Running 0 5d 10.1.30.15 test-kube02 kube-proxy-mx057 1/1 Running 0 5d 10.1.30.16 test-kube03 kube-scheduler-test-kube01 1/1 Running 0 5d 10.1.30.14 test-kube01 kubernetes-dashboard-2039414953-65392 1/1 Running 0 5d 10.244.2.2 test-kube03 monitoring-grafana-3975459543-c9pl1 1/1 Running 0 5d 10.244.3.3 test-kube02 monitoring-influxdb-3480804314-lnrr3 1/1 Running 0 5d 10.244.3.4 test-kube02 nginx-ingress-controller-429549519-q8jvl 1/1 Running 0 4d 10.1.30.15 test-kube02 weave-scope-agent-6chk2 1/1 Running 0 4d 10.1.30.16 test-kube03 weave-scope-agent-dvgz7 1/1 Running 0 4d 10.1.30.15 test-kube02 weave-scope-agent-zqpfb 1/1 Running 0 4d 10.1.30.14 test-kube01 weave-scope-app-879505699-4lwx9 1/1 Running 0 4d 10.244.2.8 test-kube03
dns test with kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
@Beanking77
added the iptables rules on master (server-1
). But on my servers, no firewalld is running (see list below of enabled services) and no such file : /usr/lib/sysctl.d/00-system.conf
sudo ufw status
Status: inactive
It is not working better but I am probably missing something (I am a newbie on linux networking config).
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp dpt:9898
ACCEPT tcp -- anywhere anywhere tcp dpt:6443
ACCEPT tcp -- anywhere anywhere tcp dpt:8472
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
...
ubuntu@server-1:~$ systemctl list-unit-files | grep enabled
acpid.path enabled
accounts-daemon.service enabled
apparmor.service enabled
atd.service enabled
autovt@.service enabled
cloud-config.service enabled
cloud-final.service enabled
cloud-init-local.service enabled
cloud-init.service enabled
console-setup.service enabled
cron.service enabled
dbus-org.freedesktop.resolve1.service enabled
docker.service enabled
ebtables.service enabled
friendly-recovery.service enabled
getty@.service enabled
iscsi.service enabled
iscsid.service enabled
keyboard-setup.service enabled
kubelet.service enabled
lvm2-monitor.service enabled
lxcfs.service enabled
lxd-containers.service enabled
networking.service enabled
ondemand.service enabled
open-iscsi.service enabled
open-vm-tools.service enabled
pollinate.service enabled
resolvconf.service enabled
rsyslog.service enabled
setvtrgb.service enabled
snapd.autoimport.service enabled
snapd.service enabled
snapd.system-shutdown.service enabled
ssh.service enabled
sshd.service enabled
syslog.service enabled
systemd-resolved.service enabled
systemd-timesyncd.service enabled
ufw.service enabled
unattended-upgrades.service enabled
ureadahead.service enabled
acpid.socket enabled
apport-forward.socket enabled
dm-event.socket enabled
docker.socket enabled
lvm2-lvmetad.socket enabled
lvm2-lvmpolld.socket enabled
lxd.socket enabled
snapd.socket enabled
uuidd.socket enabled
remote-fs.target enabled
apt-daily.timer enabled
snapd.refresh.timer enabled
@ronanquillevere According to https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ at limitation section, only RHEL/CentOS 7 user might to edit that file (I'm CentOS 7.3).
Have you test your network, like test dns with kubectl exec -ti busybox -- nslookup kubernetes.default
.
Make sure network should work properly and the dashboard should work on minion.
@Beanking77 @floreks I am looking at that right now, try to fix my DNS :)
kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
But everything looks fine
kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local local
options ndots:5
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
kube-dns-692378583-5zj21 3/3 Running 0 17h
I got not error/warnings in the logs of kubedns
and dnsmasq
. I do not have the healthz container but I believe this is not an issue.
And my DNS service is up an exposing endpoins
ubuntu@server-1:~$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 17h
kubernetes-dashboard 10.97.135.242 <none> 80/TCP 1h
ubuntu@server-1:~$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 10.244.0.4:53,10.244.0.4:53 17h
Created a new issue https://github.com/kubernetes/dns/issues/118
@ronanquillevere In my experienced in CentOS, it might be firewall or ip table rule issue. But I'm not quite sure if it happen in ubuntu, too. It might be related to system/firewall.
I have a similar issue with kubernetes dashboard. I have used kubeadm for k8s install and then installed dashboard using kubectl apply -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
. But from a long time ( more than 40 minutes) the state is still in ContainerCreating. I don't seem to understand the error. Can someone help me with this please.
When i try to check the logs with kubectl logs --namespace=kube-system kubernetes-dashboard-xxxxx i get Error from server (Bad Request): container "kubernetes-dashboard" in pod "kubernetes-dasshboard-xxx" is waiting to start: ContainerCreating
@floreks i did install busybox as per the link you mentioned. But even there the pod busybox remains in pending state. Could you please suggest on how I can proceed? With this i get pod busybox does not have a host assigned
Did you taint the node to allow application deployment on master?
kubectl taint nodes --all node-role.kubernetes.io/master-
@floreks it gives error: at least one taint update is required
The instead of --all
specify node name you want to taint.
@floreks I get the same error with kubectl taint nodes "master_node_name" node-role.kubernetes.io/master
. I am a newbie to K8S. So could you please elaborate
Did you forgot -
sign at the end of command?
Oh i didn't notice that. Now i get node "master-node-name" untainted
. Now after running the kube-proxy
command i get 503 service unavailable error
This is not a dashboard issue but cluster. As you have used kubeadm please try to ask for help on https://github.com/kubernetes/kubeadm or https://github.com/kubernetes/kubernetes. We are dealing here only with dashboard specific problems.
Issue details
Unable to access dashboard on http://master_ip/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Message "no endpoints available for service "kubernetes-dashboard""
I did the steps given in http://kubernetes.io/docs/user-guide/ui-access/, but still no result.
When using v0.19.3, I was able to access the dashboard.
Observed result
Unable to access UI