dashboard on none: unexpected response code: 503

irrgit commented 5 years ago

I cannot get minikube dashboard to work. Possible related issue #2380. Minikube was started with these options:

sudo -E  minikube start --vm-driver=none --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf --extra-config=kubelet.cgroup-driver=systemd

kube@kube:~$ minikube version
minikube version: v1.1.0

kube@kube:~$ kubectl get pods -n kube-system kubernetes-dashboard-5f7b999d65-jmv67 
NAME                                    READY   STATUS    RESTARTS   AGE
kubernetes-dashboard-5f7b999d65-jmv67   1/1     Running   1          15d

kube@kube:~$ kubectl get svc -n kube-system kubernetes-dashboard 
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes-dashboard   ClusterIP   10.97.218.152   <none>        80/TCP    15d

kube@kube:~$ kubectl get deployments -n kube-system kubernetes-dashboard 
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-dashboard   1/1     1            1           15d

kube@kube:~$ sudo -E minikube dashboard --logtostderr --v=2
I0524 19:19:50.398017   11405 none.go:231] checking for running kubelet ...
I0524 19:19:50.398044   11405 exec_runner.go:39] Run: systemctl is-active --quiet service kubelet
* Enabling dashboard ...
I0524 19:19:50.404894   11405 none.go:231] checking for running kubelet ...
I0524 19:19:50.404906   11405 exec_runner.go:39] Run: systemctl is-active --quiet service kubelet
* Verifying dashboard health ...
I0524 19:19:50.448181   11405 service.go:225] Found service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes-dashboard,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/services/kubernetes-dashboard,UID:75c9cad3-7259-11e9-9cee-9829a6fadae6,ResourceVersion:1674809,Generation:0,CreationTimestamp:2019-05-09 12:53:36 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,app: kubernetes-dashboard,kubernetes.io/minikube-addons: dashboard,kubernetes.io/minikube-addons-endpoint: dashboard,},Annotations:map[string]string{kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"kubernetes-dashboard","kubernetes.io/minikube-addons":"dashboard","kubernetes.io/minikube-addons-endpoint":"dashboard"},"name":"kubernetes-dashboard","namespace":"kube-system"},"spec":{"ports":[{"port":80,"targetPort":9090}],"selector":{"app":"kubernetes-dashboard"}}}
,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 80 {0 9090 } 0}],Selector:map[string]string{app: kubernetes-dashboard,},ClusterIP:10.97.218.152,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}
* Launching proxy ...
I0524 19:19:50.448629   11405 dashboard.go:139] Executing: /usr/local/bin/kubectl [/usr/local/bin/kubectl --context minikube proxy --port=0]
I0524 19:19:50.449049   11405 dashboard.go:144] Waiting for kubectl to output host:port ...
I0524 19:19:50.506108   11405 dashboard.go:162] proxy stdout: Starting to serve on 127.0.0.1:40167
* Verifying proxy health ...
I0524 19:19:50.524408   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:50 GMT]] Body:0xc0004b9600 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc000406400 TLS:<nil>}
I0524 19:19:50.524546   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:51.524706   11405 utils.go:114] retry loop 1
I0524 19:19:51.528851   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:51 GMT]] Body:0xc0002e4680 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011d600 TLS:<nil>}
I0524 19:19:51.528977   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:52.529131   11405 utils.go:114] retry loop 2
I0524 19:19:52.534004   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:52 GMT]] Body:0xc0002e4740 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011dd00 TLS:<nil>}
I0524 19:19:52.534103   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:53.534307   11405 utils.go:114] retry loop 3
I0524 19:19:53.539371   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:53 GMT]] Body:0xc0004b9900 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011de00 TLS:<nil>}
I0524 19:19:53.539473   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s

kube@kube:~$ sudo -E minikube dashboard --url
* Enabling dashboard ...
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
X http://127.0.0.1:34183/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
...
kube@kube:~$

OS: CentOS Linux release 7.6.1810 (Core)

winds6206 commented 5 years ago

Hi

I am encounter the same problem with --vm-driver=none

CentOS 7 minimal install

root@minikube[/tmp/Minikube]# minikube dashboard
* Enabling dashboard ...
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
X http://127.0.0.1:33959/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
...

root@minikube[/tmp/Minikube]# minikube version
minikube version: v1.1.0

lianghuiyuan commented 5 years ago

+1 Ubuntu 18.04
$ minikube version minikube version: v1.1.0

$ sudo minikube start --image-mirror-country=cn
  ......

$ sudo minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
💣  http://127.0.0.1:40339/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
......

NovikovEvgeny commented 5 years ago

I had the same issue

But managed to make it work using --docker-env parameters and setting HTTP_PROXY, HTTPS_PROXY and NO_PROXY env variables as proposed here

minikube stop
minikube delete
rm -rf  ~/.minikube
minikube start --vm-driver=none --docker-env HTTP_PROXY=http://myproxy.com:8080 --docker-env HTTPS_PROXY=http://myproxy.com:8080 --docker-env NO_PROXY=127.0.0.1

Ubuntu 16.04 minikube 1.1.0

zosocanuck commented 5 years ago

Also having the same issue on CentOS 7.6 with minikube 1.1.0:

minikube start --vm-driver=none --extra-config=apiserver.authorization-mode=RBAC

minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
💣  http://127.0.0.1:36410/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503

UPDATE Managed to solve this by following suggest in this link

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

irrgit commented 5 years ago

Here are the logs for the pod of the dashboard failing. I guess after trying to enable it too many times, it has entered a failed state now.

kube@kube:~$ kubectl logs -f kubernetes-dashboard-79dd6bfc48-rbll6 -n kube-system
2019/06/05 20:20:33 Starting overwatch
2019/06/05 20:20:33 Using in-cluster config to connect to apiserver
2019/06/05 20:20:33 Using service account token for csrf signing
2019/06/05 20:20:33 Successful initial request to the apiserver, version: v1.14.2
2019/06/05 20:20:33 Generating JWE encryption key
2019/06/05 20:20:33 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2019/06/05 20:20:33 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2019/06/05 20:20:33 Synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: unexpected object: &Secret{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Data:map[string][]byte{},Type:,StringData:map[string]string{},}
2019/06/05 20:20:33 Storing encryption key in a secret
panic: secrets is forbidden: User "system:serviceaccount:kube-system:default" cannot create resource "secrets" in API group "" in the namespace "kube-system"

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.(*rsaKeyHolder).init(0xc42040a2e0)
    /home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:131 +0x35e
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.NewRSAKeyHolder(0x1367500, 0xc42046e180, 0xc42046e180, 0x1213a6e)
    /home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:170 +0x64
main.initAuthManager(0x13663e0, 0xc420520120, 0xc4204dbcd8, 0x1)
    /home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:185 +0x12c
main.main()
    /home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:103 +0x26b

anibalardid commented 5 years ago

I fixed issue deleting .kube and .minikube directories from my home , maybe it happens because I previously testing using virtualbox let some problems there.

try it ;)

renhe134880 commented 5 years ago

I fixed issue . just: minikube stop minikube start

cc189 commented 5 years ago

resolved follow Minikube RBAC Quick Start https://gist.github.com/F21/08bfc2e3592bed1e931ec40b8d2ab6f5

tstromberg commented 5 years ago

@irrgit - I believe this issue is now addressed by minikube v1.4, as it upgrades the dashboard to 2.0 and uses a different RBAC configuration. If you still see this issue with minikube v1.4 or higher, please reopen this issue by commenting with /reopen

Thank you for reporting this issue!

GarretSidzaka commented 5 years ago

/reopen when i run it sometimes hangs forever and sometimes throws 503

garretsidzaka@cloudstack:/usr/bin$ minikube version minikube version: v1.4.0 commit: 7969c25a98a018b94ea87d949350f3271e9d64b6 ubuntu vm 1804 vm-driver=none

garretsidzaka@cloudstack:~$ sudo minikube dashboard

Verifying dashboard health ...
Launching proxy ...
Verifying proxy health ...

^C garretsidzaka@cloudstack:~$

HakNinja commented 1 week ago

I encountered the same issue on Ubuntu 24.04 LTS, but the following steps worked for me:


minikube stop
minikube delete
rm -rf ~/.minikube
minikube start

kubernetes / minikube

dashboard on none: unexpected response code: 503 #4352