search

conjure-up / spells

conjure-up spells registry
79 stars 37 forks source link

Got "502 Bad Gateway" issue with Nginx with canonical-kubernetes spell while running addon #155

Open shrajfr12 opened 6 years ago

shrajfr12 commented 6 years ago

Running conjure-up from edge channel for canonical-kubernetes spell gave the following error while executing a kubectl command to create a storage class in the addon after-deploy script.

Error from server (InternalError): an error on the server ("\r\n502 Bad Gateway\r\n<body bgcolor=\"white\">\r\n

502 Bad Gateway

\r\n
nginx/1.10.3 (Ubuntu)
\r\n\r\n") has prevented the request from succeeding

the same command execute fine on the command line.

Cynerva commented 6 years ago

This is a reply from kubeapi-load-balancer indicating that kube-apiserver is unreachable.

This happens normally during cluster deployment, and also briefly when changing certain config options on the charm. We can't really get around the fact that kube-apiserver has to be restarted occasionally; however, ideally, the cluster should be in a settled state while addon code is running.

In this case, I suspect that the restart was caused by a config change to kubernetes-master in the enable-cni step: https://github.com/conjure-up/spells/blob/0da9c9911f6c07a65e9402af381e3aaaf42e8426/canonical-kubernetes/steps/04_enable-cni/ec2/enable-cni#L239-L241

Would it make sense for the enable-cni step to wait for everything to settle after changing the config? I think so.