Fix python2 build, allow pure python3 build for rhel8, update packages

[juliusvonkohout]

Update everything to python3 and do not use python2 in any way. This greatly simplifies the assemble script and makes it ready for rhel8/centos8 python36 containers which ship only python3. Furthermore Python2 is EOL 2020.

[GrahamDumpleton]

Is there a specific reason for needing to do this? That is, something that doesn't work without it? It will remove the current ability that still exists, to build the notebook image for Python 2.7.

[juliusvonkohout]

As i pointed out in the first post, it is necessary for rhel8/centos8 python36 containers which ship only python3. I made those changes such that it builds with the freely available rhel8 ubi python36 image. For centos8 it will be the same.

Furthermore Python2 is EOL 2020 and packages like supervisor dropped Python 2 support in their latest and safest versions. More packages will follow that route.

And in addition changing site-packages location does not work with the previously used virtualenv approach.

So there are 3 severe reasons and the benefit of way less code and complexity.

[GrahamDumpleton]

As much as Python 2 is EOL in 2020, that is only as far as the PSF is concerned. Reality is that Red Hat will continue to support it for paying customers so long as prior RHEL versions with Python 2 are still supported. It isn't possible therefore to completely drop Python 2 support. At least not just yet, because enterprise organisations, don't necessarily upgrade things that quickly.

Any changes made to support changes in RHEL 8 around the switch to Python 3, will need to be done in a way for now that still allows the existing CentOS Python 2.7 base image to be used when running S2I builds. Thus you can't change pip to pip3. If virtualenv has been changed in an incompatible way such that pip no longer exists in the virtual environment, then the script will need to auto detect which to run so it still works with Python 2.7. Similarly, with supervisor, it will still need to install it, if is detected that is Python 2.7 and there is no system package for supervisor installed.

So although understand what you want to achieve, it will need to be done in a way that still supports Python 2.7 for now. When I work out where the UBI image is I'll give it a go and look at what changes can be made.

[juliusvonkohout]

The ubi image is here https://access.redhat.com/containers/?tab=images#/registry.access.redhat.com/ubi8/python-36. It does not have the plain python command anymore, but python3.

I just noticed that there are not any direct python2/3 calls in the assemble script anymore, so we can keep all good changes but have to replace pip3 with pip, which should just work as before.

So at the moment we do not even need this:

    if hash python3 2>/dev/null; then
        alias python='python3' 
    else
        alias python='python2' 
    fi

Of course jupyterlab wont run on python2 (only jupyter notebook) as it was before my changes.

I already updated my pull request.

[juliusvonkohout]

I did some test with python 2. As you can see python2 build is broken for a long time anyway. But my changes are not the reason. As you can see below a lot of packages dropped python2 support e.g jupyterhub > 0.7.2, jupyterlab > 0.33.12 etc.

You would have to use 
    if hash python3 2>/dev/null; then
        pip install -r /tmp/src/requirements.txt
    else
        pip install -r /tmp/src/requirements_python2.txt
    fi

and create the file /tmp/src/requirements_python2.txt for python2.

Cloning "https://github.com/juliusvonkohout/jupyter-notebooks.git " ...
    Commit: f57b031fdf673d266e6db58edf1dfd8685974cf7 (Update assemble)
    Author: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
    Date:   Sat May 11 03:33:38 2019 +0200
Using centos/python-27-centos7@sha256:1504f788f697a579469b3966396e676049078a5ff8815b3bc2ab5377c726f41e as the s2i builder image
You are using pip version 7.1.0, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip
Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl  (1.4MB)
Collecting setuptools
Downloading https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl  (575kB)
Collecting wheel
Downloading https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl 
Installing collected packages: pip, setuptools, wheel
Found existing installation: pip 7.1.0
Uninstalling pip-7.1.0:
Successfully uninstalled pip-7.1.0
Found existing installation: setuptools 18.0.1
Uninstalling setuptools-18.0.1:
Successfully uninstalled setuptools-18.0.1
Found existing installation: wheel 0.24.0
Uninstalling wheel-0.24.0:
Successfully uninstalled wheel-0.24.0
Successfully installed pip-19.1.1 setuptools-41.0.1 wheel-0.33.4
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting mod_wsgi==4.6.5
Downloading https://files.pythonhosted.org/packages/47/69/5139588686eb40053f8355eba1fe18a8bee94dc3efc4e36720c73e07471a/mod_wsgi-4.6.5.tar.gz  (490kB)
Building wheels for collected packages: mod-wsgi
Building wheel for mod-wsgi (setup.py): started
Building wheel for mod-wsgi (setup.py): finished with status 'done'
Stored in directory: /tmp/pip-ephem-wheel-cache-khjUhO/wheels/28/f2/06/a35ecac864a04a754d59ea59e44bf92ba4f50b2c1857491186
Successfully built mod-wsgi
Installing collected packages: mod-wsgi
Successfully installed mod-wsgi-4.6.5
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting notebook==5.7.4 (from -r /tmp/src/requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/f8/a6/dbdf0954d073ab2cf5421cad20334d1e4c780da37205c04522c045298f67/notebook-5.7.4-py2.py3-none-any.whl  (9.0MB)
Collecting jupyterhub==0.9.4 (from -r /tmp/src/requirements.txt (line 2))
  ERROR: Could not find a version that satisfies the requirement jupyterhub==0.9.4 (from -r /tmp/src/requirements.txt (line 2)) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.6.1, 0.7.0b1, 0.7.0, 0.7.1, 0.7.2)
ERROR: No matching distribution found for jupyterhub==0.9.4 (from -r /tmp/src/requirements.txt (line 2))
error: build error: non-zero (13) exit code from centos/python-27-centos7@sha256:1504f788f697a579469b3966396e676049078a5ff8815b3bc2ab5377c726f41e

[juliusvonkohout]

I did the necessary changes and built the minimal notebook with Built with centos/python-27-centos7 and rhscl/python-36-rhel7. Jupyterlab and notebook is working fine. Please merge.

[GrahamDumpleton]

Before it can be merged I need to evaluate how this impacts the separate JupyterHub repo and image and whether changes are going to need to be made to it, due to the JupyterHub integration. The JupyterHub repo and image don't use JupyterHub 1.0, and changing that will likely have some flow on affects to a number of things based on the JupyterHub repo and image. For example, they drop some support for backwards compatibility with configuration file variable naming. Various POC examples based on the JupyterHub repo and image may need to be updated as a result.

Also looking again at issue of supporting Python 2.7. Looking at how you are trying to accomodate that, may yet be easier to drop support for Python 2.7, so having a further think about that first.

I have some other changes I need to make to the notebook repo to support a way of handling WebDav access when run under JupyterHub, so when I do that I will look at integrating the changes. I may replicate changes as I do this other work, rather than merge as is.

[juliusvonkohout]

Well i can revert it to jupyterhub==0.9.4 then you just have to decide if you want python2 support. If you dont want python2, i will just remove the if clause and you can merge and tag this.

Afterwards you can deal with webdav and jupyterhub 1.0 without me interfering.

[GrahamDumpleton]

There is more than just that.

One doesn't need separate requirements.txt files and instead one can use environment markers (https://www.python.org/dev/peps/pep-0508/#environment-markers) for stating what versions of packages should be used for different Python versions.

The creation of the /opt/app-root/data directory was removed. That directory exists as a de-facto place to store databases etc, with potential to mount a persistent volume on it for persistence.

And supervisor is now installed into the virtual environment that Jupyter notebook uses, rather than being separate. The supervisor package only seems to depend on meld3 which is perhaps unlikely to be used by a user, but still safer to install supervisor in its own virtual environment to avoid risk that user installing incompatible versions of supervisor or meld3 in their requirements.txt will break the ability for the image to start up where supervisor is used. Will need to think about how big of an issue that one really is though.

[GrahamDumpleton]

You can hold off doing anything further. I have pushed a bunch of changes on develop branch which addresses everything, with tweaks as to how it was done. This PR was created against master branch because default branch in GitHub hadn't been set correctly to develop, which have now fixed. So maybe try develop branch and see whether it works with the ubi image.

[juliusvonkohout]

Thank you very much. Just one comment. i noticed the line

JUPYTERLAB_HUB_EXTENSION_VERSION=${JUPYTERLAB_HUB_EXTENSION_VERSION:-0.12.0}

in https://github.com/jupyter-on-openshift/jupyter-notebooks/blob/7ed55ccc3964e2a541ac9be6b50dbcf82851cbbc/minimal-notebook/.s2i/bin/assemble#L27

You can remove the pinned version. it will always install the extension version to the corresponding jupyterlab. At the momenen 0.12.0 and if you upgrade to jupyterlab 1.0.0a3 it uses a newer version. So you can remove "JUPYTERLAB_HUB_EXTENSION_VERSION=${JUPYTERLAB_HUB_EXTENSION_VERSION:-0.12.0} "

entirely and replace

" jupyter labextension install \ @jupyterlab/hub-extension@^${JUPYTERLAB_HUB_EXTENSION_VERSION}"

in line 34/35 with

"jupyter labextension install @jupyterlab/hub-extension"

This will be necessary anyway if you upgrade to jupyterlab 1.0

Nevertheless i am closing this pull request and hope that you make this little change and/or provide a new release tag soon. Thank you very much.

[GrahamDumpleton]

I presume that must be a new thing. Back in time if I didn't pin it, I would see issues as it was always installing the latest, which caused problems when you hadn't updated jupyterlab and it required an older version. The extension is pretty stable now though, so may not change. Can you point though at something which says it will use the appropriate version by itself?

[juliusvonkohout]

I just tested it myself. it was in this pull request here(branch now deleted). I am able to build jupyterl 0.35.6 and 1.0.0a3 without specifying the extension version. I did builds for both versions and tested them. if you use jupyterlab==0.35.6 the extension version 1.0.0-alpha.6 is checked and seen as incompatible and 0.12.0 is installed instead. If you use jupyterlab==1.0.0a3 the 1.0.0-alpha.6 version is installed. Just try it out yourself on your laptop or see my install logs. 0.35.6.txt 1.0.0a3.txt