search

wildtreetech / ohjh

ohjh - The OpenHumans JupyterHub deployment
MIT License
8 stars 7 forks source link

Upgrade to newer JHub helm chart #29

Closed betatim closed 6 years ago

betatim commented 6 years ago

This upgrades the hub to the latest available chart version.

We had a failed deploy because at first the pdb (pod disruption budget) was enabled which caused a failed deploy.

After fixing that in values.yaml helm upgrade ... failed with an error like Error: UPGRADE FAILED: no ServiceAccount with the name "autohttps" found. This was probably because the failed deploy left things in a "weird" state. The solution was to helm rollback ohjh 79 where 79 was the revision before the upgrade. After that helm upgrade ... succeeded.

We ended up with a crashing hub pod though because the proxy rolebinding(?) did not have the right permissions to list all the ingresses.

We solved that by deleting the deployment and installing from scratch.

consideRatio commented 6 years ago

Thanks for writing this down!

A helm bug causing multiple revision of a release to be considered "deployed" has caused a lot of troubles like this. Instead of deleting the full chart, one can do some manual cleanup to ensure only one release is considered Deployed at the same time.

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/638

Also, some changes require --force command as the resource cannot be patched but must be recreated, then --force is needed on the helm upgrade. Example, you change a deployments pod selector field or a PDB.