Runtime error when running jupyterhub-deployer

[willingc]

Hi @GrahamDumpleton,

I'm working through this project so I can add references to it from our JupyterHub docs. I came across this runtime error while running on an openshift online starter instance. Any ideas of the cause? Thanks.

[GrahamDumpleton]

What does oc version give. This looks more like could be a version incompatibility. Ensure you have same oc client versions as OpenShift environment.

[GrahamDumpleton]

Also, although you will likely be able to start JupyterHub on OpenShift Online Starter, you will not be able to launch a notebook as you don't have enough memory resources. To be able to run on Online Starter, would be necessary to reduce the memory allocated to PostrgreSQL and JupyterHub to 256MB. Together they would take 512MB and give you 512MB for one notebook.

[GrahamDumpleton]

If you sort out any oc version mismatch issue, reloaded the template:

oc replace -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/templates.json

and run:

oc new-app --template jupyterhub-deployer \
  --param JUPYTERHUB_MEMORY=256M \
  --param DATABASE_MEMORY=256M \
  --param NOTEBOOK_MEMORY=256M

I still need to update the README with details of how now can change memory, and also need to add some stuff in to allow binary units for memory, e.g., 256Mi, for notebook memory. It would be nice actually if the JupyterHub code itself allowed them, currently only allows 256M.

• https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/spawner.py#L423

It could translate any units to bytes.

I want to accept Mi because Kubernetes does and so want to avoid confusion if people use Mi units instead of M as used to doing so with Kubernetes/OpenShift.

[GrahamDumpleton]

Actually, that needs to be:

oc new-app --template jupyterhub-deployer \
  --param JUPYTERHUB_MEMORY=256Mi \
  --param DATABASE_MEMORY=256Mi \
  --param NOTEBOOK_MEMORY=268435456

The lowest you can wind down memory on OpenShift Online is 256Mi, not 256M. This illustrates why need Mi to be accepted. :-(

[GrahamDumpleton]

Lets try again:

oc new-app --template jupyterhub-deployer \
  --param JUPYTERHUB_MEMORY=256Mi \
  --param DATABASE_MEMORY=256Mi \
  --param NOTEBOOK_MEMORY=269M

JupyterHub will not convert string containing integer, with no units, to integer. So I will need to cope with converting that as well.

[GrahamDumpleton]

Unfortunately addressing issue with units in JupyterHub can now never be fixed. The JupyterHub convention has M being power of two convention, when understand standards say Mi is and M is meant to be power of 10 version.

• https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/traitlets.py#L36

You can't fix it because existing usage could break things as switching will mean less memory is then used in existing systems than was asked for.

[GrahamDumpleton]

All cleaned up now.

You will need to do:

oc replace -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyterhub-quickstart/master/templates.json

AND

oc start-build bc/jupyterhub
oc logs --follow bc/jupyterhub

When the re-build of jupyterhub is done, you can then do:

oc new-app --template jupyterhub-deployer \
  --param JUPYTERHUB_MEMORY=256Mi \
  --param DATABASE_MEMORY=256Mi \
  --param NOTEBOOK_MEMORY=256Mi

Remember, you can delete the existing instance first by running:

oc delete all,configmap,pvc,serviceaccount,rolebinding --selector app=jupyterhub

[GrahamDumpleton]

With changes to allow memory to be overridden, I have confirmed it will work on Online Starter. That environment though is very memory constrained with only 1Gi for applications and 1Gi for builds. It is only meant for experimentation, so it will show that JupyterHub can be deployed, but you will not be able to do much. Because of the constraints, there is a chance that notebooks may not spin up quick enough before the start timeout is hit. I have the default as 120 seconds. If necessary it would need to be increased by editing the config map for jupyterhub-cfg and adding:

c.Spawner.start_timeout = 180

Then trigger a new deployment of jupyterhub to have change picked up.

BTW, never set start timeout on KubeSpawner if using ProfilesSpawner. This is because JupyterHub doesn't use it and uses what is set as default on Spawner.start_timeout from which ProfilesSpawner derives. That is something that perhaps needs to be fixed in ProfilesSpawner. That is, it should somehow return the start timeout from the spawner which is selected.

[willingc]

Hi @GrahamDumpleton,

Thanks a bunch. Just a quick update: I've successfully built everything using minishift (Thank you @mhrivnak for the minishift and openshift conversation at SCALE16x yesterday).

I'm going to try this on the starter online version this week. Will keep you posted.

[GrahamDumpleton]

Okay, thanks for letting me know. I will close this issue now then.