search

hobby-kube / guide

Kubernetes clusters for the hobbyist.
MIT License
5.6k stars 259 forks source link

502 Bad Gateway → Deploying the Kubernetes Dashboard #18

Closed jujes closed 7 years ago

jujes commented 7 years ago

Hi, someone know how fix it? all services are working...

502 Bad Gateway nginx/1.13.2

Getting kubectl states....
NAME                          STATUS    AGE       VERSION
mas-00                        Ready     6h        v1.7.0
min-00                        Ready     6h        v1.7.0
vmi124777.contaboserver.net   Ready     6h        v1.7.0
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
ingress       default-http-backend-726995137-rpvsg       1/1       Running   1          5h
ingress       kube-lego-2933009699-gjb48                 1/1       Running   1          35m
ingress       nginx-ingress-controller-588775257-4gn3q   1/1       Running   2          5h
kube-system   heapster-3875886179-gbnq2                  1/1       Running   1          40m
kube-system   kube-apiserver-mas-00                      1/1       Running   1          6h
kube-system   kube-controller-manager-mas-00             1/1       Running   1          6h
kube-system   kube-dns-2425271678-xsh8f                  3/3       Running   3          6h
kube-system   kube-proxy-hkw96                           1/1       Running   1          6h
kube-system   kube-proxy-k63sl                           1/1       Running   1          6h
kube-system   kube-proxy-q6x71                           1/1       Running   1          6h
kube-system   kube-scheduler-mas-00                      1/1       Running   1          6h
kube-system   kubernetes-dashboard-4079053634-fkz3m      1/1       Running   1          40m
kube-system   weave-net-32zwn                            2/2       Running   4          6h
kube-system   weave-net-lqss1                            2/2       Running   3          6h
kube-system   weave-net-w9zsg                            2/2       Running   3          6h
NAMESPACE     NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
ingress       default-http-backend       1         1         1            1           5h
ingress       kube-lego                  1         1         1            1           35m
ingress       nginx-ingress-controller   1         1         1            1           5h
kube-system   heapster                   1         1         1            1           40m
kube-system   kube-dns                   1         1         1            1           6h
kube-system   kubernetes-dashboard       1         1         1            1           40m
NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes             10.96.0.1        <none>        443/TCP         6h
ingress       default-http-backend   10.110.68.42     <none>        80/TCP          5h
ingress       kube-lego-nginx        10.108.198.188   <none>        8080/TCP        35m
kube-system   heapster               10.100.5.228     <none>        80/TCP          40m
kube-system   kube-dns               10.96.0.10       <none>        53/UDP,53/TCP   6h
kube-system   kubernetes-dashboard   10.97.124.69     <none>        80/TCP          40m

thanks in advance

pstadler commented 7 years ago

You should get some more insights by checking the logs of nginx, e.g. 

kubectl logs --follow -n ingress deployment/nginx-ingress-controller

I just recently deployed a Kubernetes 1.7.0 cluster and exposed the dashboard using the nginx ingress w/o any problems.

jujes commented 7 years ago

Hi @pstadler, yes here the logs:

[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 7.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
I0712 08:43:31.266384       7 launch.go:105] &{NGINX 0.9.0-beta.10 git-9ef5330 https://github.com/kubernetes/ingress}
I0712 08:43:31.266481       7 launch.go:108] Watching for ingress class: nginx
I0712 08:43:31.268744       7 launch.go:262] Creating API server client for https://10.96.0.1:443
I0712 08:43:31.270942       7 nginx.go:182] starting NGINX process...
I0712 08:43:31.322172       7 launch.go:124] validated ingress/default-http-backend as the default backend
I0712 08:43:31.456314       7 controller.go:1183] starting Ingress controller
I0712 08:43:31.561959       7 leaderelection.go:203] attempting to acquire leader lease...
W0712 08:43:31.562177       7 controller.go:576] service ingress/default-http-backend does not have any active endpoints
I0712 08:43:31.668879       7 leaderelection.go:213] successfully acquired lease ingress/ingress-controller-leader-nginx
I0712 08:43:33.847816       7 controller.go:427] backend reload required
I0712 08:43:33.853447       7 metrics.go:34] changing prometheus collector from  to vts
I0712 08:43:34.028831       7 controller.go:437] ingress backend successfully reloaded...
I0712 08:43:34.657473       7 controller.go:427] backend reload required
I0712 08:43:34.862785       7 controller.go:437] ingress backend successfully reloaded...
W0712 08:47:20.604162       7 controller.go:826] error obtaining service endpoints: service kube-system/kubernetes-dashboard does not exist
I0712 08:47:20.605688       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
E0712 08:47:20.606514       7 annotations.go:87] error reading BasicDigestAuth annotation in Ingress kube-system/kubernetes-dashboard: unexpected error reading secret kube-system/kubernetes-dash
board-auth: secret kube-system/kubernetes-dashboard-auth was not found
I0712 08:47:20.608118       7 controller.go:427] backend reload required
I0712 08:47:20.619549       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"b77a9dd7-66de-11e7-a958-de2b4806f006", APIVersion:
"extensions", ResourceVersion:"39843", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress kube-system/kubernetes-dashboard
I0712 08:47:21.059893       7 controller.go:437] ingress backend successfully reloaded...
W0712 08:47:21.524269       7 controller.go:876] service kube-system/kubernetes-dashboard does not have any active endpoints
I0712 08:47:21.525273       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 08:47:21.527464       7 controller.go:427] backend reload required
W0712 08:47:21.590050       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 08:47:21.717868       7 controller.go:437] ingress backend successfully reloaded...
I0712 08:47:21.736260       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress", Name:"kube-lego-nginx", UID:"b82a4b1e-66de-11e7-a958-de2b4806f006", APIVersion:"extensio
ns", ResourceVersion:"39863", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ingress/kube-lego-nginx
I0712 08:47:24.658624       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 08:47:24.662491       7 controller.go:427] backend reload required
I0712 08:47:24.836419       7 controller.go:437] ingress backend successfully reloaded...
I0712 08:47:27.993161       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 08:47:31.327102       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
W0712 08:47:31.590637       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 08:47:31.655318       7 status.go:310] updating Ingress ingress/kube-lego-nginx status to [{10.8.56.141 }]
I0712 08:47:31.659537       7 status.go:310] updating Ingress kube-system/kubernetes-dashboard status to [{10.8.56.141 }]
I0712 08:47:31.738096       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress", Name:"kube-lego-nginx", UID:"b82a4b1e-66de-11e7-a958-de2b4806f006", APIVersion:"extensio
ns", ResourceVersion:"39885", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress ingress/kube-lego-nginx
I0712 08:47:31.739317       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"b77a9dd7-66de-11e7-a958-de2b4806f006", APIVersion:
"extensions", ResourceVersion:"39886", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-system/kubernetes-dashboard
I0712 08:47:34.658498       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 08:47:37.992095       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
W0712 08:47:41.593220       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 08:47:42.453655       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 08:47:44.660318       7 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
W0712 08:47:51.593586       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:01.593857       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:11.594119       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:21.594576       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:31.595169       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:41.595749       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:48:51.602713       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:01.603085       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:11.603680       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:21.604263       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:31.604706       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:41.608811       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:49:51.609082       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:50:01.609662       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
82.213.226.244 - [82.213.226.244] - - [12/Jul/2017:08:50:06 +0000] "GET / HTTP/1.1" 401 597 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safar
i/537.36" 391 0.000 [kube-system-kubernetes-dashboard-http] - - - -
W0712 08:50:11.610107       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
82.213.226.244 - [82.213.226.244] - jujes [12/Jul/2017:08:50:17 +0000] "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 S
afari/537.36" 434 3.150 [kube-system-kubernetes-dashboard-http] 10.44.0.29:9090 0 3.150 502
2017/07/12 08:50:17 [error] 316#316: *6 connect() failed (113: No route to host) while connecting to upstream, client: 82.213.226.244, server: dashboard.stackup.codes, request: "GET / HTTP/1.1",
 upstream: "http://10.44.0.29:9090/", host: "dashboard.stackup.codes"
2017/07/12 08:50:20 [error] 316#316: *6 connect() failed (113: No route to host) while connecting to upstream, client: 82.213.226.244, server: dashboard.stackup.codes, request: "GET /favicon.ico
 HTTP/1.1", upstream: "http://10.44.0.29:9090/favicon.ico", host: "dashboard.stackup.codes", referrer: "http://dashboard.stackup.codes/"
82.213.226.244 - [82.213.226.244] - jujes [12/Jul/2017:08:50:20 +0000] "GET /favicon.ico HTTP/1.1" 502 575 "http://dashboard.stackup.codes/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (
KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" 411 2.968 [kube-system-kubernetes-dashboard-http] 10.44.0.29:9090 0 2.968 502
W0712 08:50:21.610581       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:50:31.611027       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:50:41.614710       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 08:50:51.615065       7 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
pstadler commented 7 years ago

Try deleting the pods of the dashboard and nginx ingress controller deployments. Don't delete them simultaneously, but one pod after another. I was running into similar issues when either nginx or kube-lego pods weren't fully deployed.

You should also check the logs of the dashboard deployment to make sure everything is working correctly.

jujes commented 7 years ago

The dashboard and nginx ingress controller deployments was deleted one by one, waiting each one are recreated

dashboad pods: nothing logs show me the command: kubectl logs --follow -n kube-system deployment/kubernetes-dashboard

instead I look into the container and get this:

Using HTTP port: 8443
Creating API server client for https://10.96.0.1:443
Successful initial request to the apiserver, version: v1.7.0
Creating in-cluster Heapster client
Using service account token for csrf signing

nginx ingress controller pods continue showing the same issue:

[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 6.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
I0712 09:43:54.308872       6 launch.go:105] &{NGINX 0.9.0-beta.10 git-9ef5330 https://github.com/kubernetes/ingress}
I0712 09:43:54.309185       6 launch.go:108] Watching for ingress class: nginx
I0712 09:43:54.311876       6 launch.go:262] Creating API server client for https://10.96.0.1:443
I0712 09:43:54.350242       6 launch.go:124] validated ingress/default-http-backend as the default backend
I0712 09:43:54.394542       6 controller.go:1183] starting Ingress controller
I0712 09:43:54.446678       6 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress", Name:"kube-lego-nginx", UID:"b82a4b1e-66de-11e7-a958-de2b4806f006", APIVersion:"extensio
ns", ResourceVersion:"39885", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ingress/kube-lego-nginx
I0712 09:43:54.454244       6 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"b77a9dd7-66de-11e7-a958-de2b4806f006", APIVersion:
"extensions", ResourceVersion:"39886", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress kube-system/kubernetes-dashboard
I0712 09:43:54.502297       6 leaderelection.go:203] attempting to acquire leader lease...
W0712 09:43:54.503211       6 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 09:43:55.419142       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 09:43:55.422859       6 controller.go:427] backend reload required
I0712 09:43:55.423522       6 metrics.go:34] changing prometheus collector from  to vts
I0712 09:43:55.619397       6 controller.go:437] ingress backend successfully reloaded...
I0712 09:43:57.686164       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 09:44:01.019366       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 09:44:04.354583       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
W0712 09:44:04.505682       6 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 09:44:07.686165       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 09:44:11.020725       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
I0712 09:44:14.352826       6 controller.go:1060] ssl certificate "kube-system/kubernetes-dashboard-tls" does not exist in local store
W0712 09:44:14.507638       6 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
W0712 09:44:24.508095       6 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
I0712 09:44:30.264139       6 leaderelection.go:213] successfully acquired lease ingress/ingress-controller-leader-nginx
W0712 09:44:34.508588       6 backend_ssl.go:46] error obtaining PEM from secret kube-system/kubernetes-dashboard-tls: secret named kube-system/kubernetes-dashboard-tls does not exist
pstadler commented 7 years ago

It looks like you're unable to get an SSL certificate for the dashboard ingress. Check out the logs of kube-lego or simply disable tls for the dashboard.

jujes commented 7 years ago

Tried both of them and not get success... even I use other 3 different domain discard some blocked status from letsencrypt... Can I send you some credentials for look around?

pstadler commented 7 years ago

Make sure your config map is correct: https://github.com/hobby-kube/manifests/blob/master/ingress/tls/configmap.yml

jujes commented 7 years ago

Yes is correct, I changed the mail: "admin@example.com"

jujes commented 7 years ago

@pstadler seems to be the ingress pack don't work on my cluster.. I'm trying a lot of configurations :)

Can you help me transferring the values from manifests/ingress/ → values.yaml file, try set up with that too?

The official nginx-ingress helm chart supports NodePort as a Service type.