search

jupyter-on-openshift / jupyter-notebooks

OpenShift compatible S2I builder for basic notebook images.
Apache License 2.0
54 stars 111 forks source link

Base image with python version equal or greater than 3.7 #25

Open jonDel opened 4 years ago

jonDel commented 4 years ago

From now, the last version of the minimal notebook is based on python 3.6, s2i-minimal-notebook-py36 How about a s2i-minimal-notebook-py37, s2i-minimal-notebook-py38, etc? I tried to search the s2i-minimal-notebook-py36 Dockerfile for building my own container with python 3.8, but couldn't find it. Any clues here?

GrahamDumpleton commented 4 years ago

Use the Jupyter Project images:

jonDel commented 4 years ago

I tried the tutorial using the official images from jupyter project, but that did not work. The notebook container did not go up, complaining that some shelI script is not present. I then ran containers from both images, s2i-minimal-notebook-py36 and jupyter/minimal-notebook and took a peek inside them. It saw that indeed, the jupyter folder is very different comparing s2i-minimal-notebook-py36 and jupyter/minimal-notebook. A lot of files missing from jupyter/minimal-notebook that breaks the deployment.

GrahamDumpleton commented 4 years ago

You are not providing sufficient information about what you are trying to do. If you are trying to use the Jupyter project images as an S2I builder with the mechanism described with this repository, you can't. For using S2I with Jupyter project images see:

Also see:

for general details of standalone deployment of Jupyter project images on OpenShift.

The link I provided above previously is specifically related to using the Jupyter project images with JupyterHub on OpenShift. I provided that since had to guess what it is you are trying to do due to the lack of details provided.

jonDel commented 4 years ago

I am trying to follow the tutorial in https://github.com/jupyter-on-openshift/jupyterhub-quickstart , more specifically using the Multi User Developer Workspace template. I do get successful using the s2i-minimal-notebook-py36 as a base image for the notebooks. But I want to use a python 3.8 base image, so I tried https://github.com/jupyter-on-openshift/jupyterhub-quickstart#using-the-jupyter-project-notebook-images, using the jupyter/minimal-notebook image with python 3,.8. However, even following the instructions:

If you still want to run the official Jupyter Project notebook images, you can, but you will need to supply additional configuration to the KubeSpawner plugin for these images to have them work. ... The special setting is supplemental_gids, with it needing to be set to include the UNIX group ID of 100.

it wont work, since there are a lot of files missing from jupyter/minimal-notebook that breaks the deployment.

GrahamDumpleton commented 4 years ago

As I already said, you need to properly explain "it wont work". I can't guess what the problem is. You need to explain properly what you added to your configuration and what errors you are getting and where.

jonDel commented 4 years ago

The event log when the hub tries to start a server with the image jupyter/minimal-notebook (https://hub.docker.com/r/jupyter/minimal-notebook) :


Your server is starting up.

You will be redirected automatically when it's ready for you.
Event log
Server requested
2020-08-25 17:51:42+00:00 [Warning] pod has unbound PersistentVolumeClaims (repeated 4 times)
2020-08-25 17:51:42+00:00 [Normal] Successfully assigned jupyterhub/jupyterhub-nb-*** to ****
2020-08-25 17:51:45+00:00 [Normal] pulling image "***/jupyter-minimal-notebook@sha256:***"
2020-08-25 17:52:21+00:00 [Normal] Successfully pulled image "***/jupyter-minimal-notebook@sha256:***"
2020-08-25 17:52:33+00:00 [Normal] Created container
2020-08-25 17:52:34+00:00 [Warning] Error: failed to start container "setup-volume": Error response from daemon: oci runtime 
  error: container_linux.go:247: 
  starting container process caused "exec: \"/opt/app-root/bin/setup-volume.sh\": 
  stat /opt/app-root/bin/setup-volume.sh: no such file or directory"
As I said, the file structure of images jupyter/minimal-notebook and s2i-minimal-notebook-py36 are very different, as jupyter/minimal-notebook uses conda. Running a container with s2i-minimal-notebook-py36 image: ``` (app-root) bash-4.2$ ls /opt/app-root/bin/ activate dask-submit jsonschema jupyter-labhub mako-render pip setup-volume.sh stop-supervisord.sh activate.csh dask-worker jupyter jupyter-migrate mod_wsgi-express pip3 start-daskscheduler.sh supervisorctl activate.fish easy_install jupyter-bundlerextension jupyter-nbconvert oc pip3.6 start-daskworker.sh supervisord activate_this.py easy_install-3.6 jupyterhub jupyter-nbextension oc-3.11 pygmentize start-gateway.sh wheel alembic echo_supervisord_conf jupyterhub-singleuser jupyter-notebook oc-4.0 pyjson5 start-kernelgateway.sh certipy iptest jupyter-kernel jupyter-run oc-4.1 python start-lab.sh chardetect iptest3 jupyter-kernelgateway jupyter-serverextension oc-4.2 python3 start-notebook.sh dask-remote ipython jupyter-kernelspec jupyter-troubleshoot oc-4.3 python3.6 start.sh dask-scheduler ipython3 jupyter-lab jupyter-trust oc-wrapper.sh python-config start-singleuser.sh dask-ssh jlpm jupyter-labextension kustomize pidproxy setup-environ.sh start-webdav.sh (app-root) bash-4.2$ ``` Running a container with jupyter/minimal-notebook image: ``` jovyan@b605b1da3c86:~$ ls /opt/app-root/bin/ ls: cannot access '/opt/app-root/bin/': No such file or directory jovyan@b605b1da3c86:~$ ls /opt/conda/bin/ 2to3 c_rehash infocmp jupyter-labhub ksu mako-render pydoc3 tclsh xz 2to3-3.8 curl-config infotocap jupyter-migrate kswitch ncurses6-config pydoc3.8 tclsh8.6 xzcat acountry curve_keygen iptest jupyter-nbconvert ktutil ncursesw6-config pygmentize tic xzcmp activate deactivate iptest3 jupyter-nbextension kvno nghttp pyjson5 tini xzdec adig derb ipython jupyter-notebook lzcat nghttpd pyjwt toe xzdiff ahost easy_install ipython3 jupyter-run lzcmp nghttpx python tput xzegrep alembic genbrk jlpm jupyter-serverextension lzdiff node python3 tqdm xzfgrep captoinfo gencfu jsonschema jupyter-troubleshoot lzegrep npm python3.8 tset xzgrep certipy gencnval jupyter jupyter-trust lzfgrep npx python3.8-config unlzma xzless chardetect gendict jupyter-bundlerextension k5srvutil lzgrep openssl python3-config unxz xzmore clear genrb jupyterhub kadmin lzless pandoc reset uuclient compile_et gss-client jupyterhub-singleuser kdestroy lzma pandoc-citeproc sclient wheel conda icu-config jupyter-kernel kinit lzmadec pip sim_client wish conda-env icuinfo jupyter-kernelspec klist lzmainfo pip3 sqlite3 wish8.6 configurable-http-proxy idle3 jupyter-lab kpasswd lzmore pkgdata sqlite3_analyzer x86_64-conda_cos6-linux-gnu-ld cph idle3.8 jupyter-labextension krb5-config makeconv pydoc tabs x86_64-conda-linux-gnu-ld ```
GrahamDumpleton commented 4 years ago

You can't use the latter recipe for persistent storage in:

with the Jupyter project images. That is, the one relying on setup-volume.sh.

For Jupyter project images, you also need to change the first recipe to:

c.KubeSpawner.user_storage_pvc_ensure = True

c.KubeSpawner.pvc_name_template = '%s-nb-{username}' % c.KubeSpawner.hub_connect_ip
c.KubeSpawner.user_storage_capacity = '1Gi'

c.KubeSpawner.volumes = [
    {
        'name': 'data',
        'persistentVolumeClaim': {
            'claimName': c.KubeSpawner.pvc_name_template
        }
    }
]

c.KubeSpawner.volume_mounts = [
    {
        'name': 'data',
        'mountPath': '/home/jovyan'
    }
]

That is, different mount point for persistent volume.

jonDel commented 4 years ago

Thank you Graham, I will give it a try and come back as soon as I have a result.

jonDel commented 4 years ago

I still got the same error. I seems the KubeSpawner always tries to execute the "setup-volume.sh" command, even when I explicitly declare "c.KubeSpawner.cmd" in the jupyterhub_config,py file.

GrahamDumpleton commented 4 years ago

It isn't c.KubeSpawner.cmd that is going to be the problem. It is going to be because you are setting c.KubeSpawner.singleuser_init_containers.

As I said above, don't use the second recipe which invokes setup-volume.sh listed in: