Pre 0.9.0 release - issue with kubespawner and user-scheduler on k8s 1.16

Boes-man commented 4 years ago

Screen Shot 2020-07-02 at 5 19 26 pm

Bug description

"Start My Server" errors with "Spawn failed"

Expected behaviour

New notebook

Actual behaviour

Error

How to reproduce

Install Pangeo: helm install pangeo/pangeo --name=pangeo --namespace=pangeo -f helm-pangeo-values.yaml

Patch kubespawner to "fix" 500 internal server error kubectl patch deploy -n pangeo hub --type json --patch '[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value": ["bash", "-c", "\nmkdir -p ~/hotfix\ncp -r /usr/local/lib/python3.6/dist-packages/kubespawner ~/hotfix\nls -R ~/hotfix\npatch ~/hotfix/kubespawner/spawner.py << EOT\n72c72\n< key=lambda x: x.last_timestamp,\n---\n> key=lambda x: x.last_timestamp and x.last_timestamp.timestamp() or 0.,\nEOT\n\nPYTHONPATH=$HOME/hotfix jupyterhub --config /srv/jupyterhub_config.py --upgrade-db\n"]}]'

Browse to Jupyter landing page Login as random user, i.e. testing123 Click "Start My Server"

Your personal set up

Kubernetes cluster on AWS (m5.4xlarge): kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:47:40Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T20:52:22Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

OS: Centos 7
Version: From Pangeo Helm Chart: ` apiVersion: v2 name: pangeo version: 0.0.1-set.by.chartpress description: An extention of jupyterhub with extra Pangeo resources dependencies:
- name: jupyterhub version: "0.9.0-beta.4.n008.hb20ad22" repository: 'https://jupyterhub.github.io/helm-chart/' import-values:
  - child: rbac parent: rbac
- name: dask-gateway Make sure to update the dask-gateway CRDs in .circleci/config.yml version: "0.7.1" repository: 'https://dask.org/dask-gateway-helm-repo/' `

https://github.com/pangeo-data/helm-chart

Thanks

manics commented 4 years ago

Hi @Boes-man, do you see this bug when using this (zero-to-jupyterhub-k8) helm chart on its own, or only when using the pangeo helm chart?

Boes-man commented 4 years ago

Thanks @manics I will give that a go. Can you please provide the instruction I should follow?

manics commented 4 years ago

Please follow the walkthrough on https://zero-to-jupyterhub.readthedocs.io/en/0.9.0/

Boes-man commented 4 years ago

Hi @manics that works... jupyter notebook starts after login. Is the pangeo install doing something wrong? Wrong versions? Thanks

consideRatio commented 4 years ago

I'm closing this in favor of https://github.com/pangeo-data/pangeo/issues/773. I think the issue is an outdated version of z2jh that didn't support k8s 1.16.

support[bot] commented 4 years ago

Hi there @Boes-man :wave:!

I closed this issue because it was labelled as a support question.

Please help us organize discussion by posting this on the http://discourse.jupyter.org/ forum.

Our goal is to sustain a positive experience for both users and developers. We use GitHub issues for specific discussions related to changing a repository's content, and let the forum be where we can more generally help and inspire each other.

Thanks you for being an active member of our community! :heart:

jupyterhub / zero-to-jupyterhub-k8s