Single conda Jupyter kernel

[Peter9192]

closes #47

• [x] Remove python kernel from kernelspec
• [x] Turn off native kernel in spawned notebooks
• [x] Update notebook dependencies/lab extension setup to use conda python instead of system python
• [x] Add conda init to user shell
• [x] Maybe rename default notebook? Currently: Python3 (Conda)

[Peter9192]

Status update:

Main jupyterhub/lab environment switched over to ewatercycle conda environment. Need to

• [x] Add packages from the era5-comparison environment
• [x] Add packages from pangeo
• [x] Try to add pymt/esmvaltool/hymuse/grpc4bmi to the default ewatercycle env rather than their own or the base env.
• [x] Clean up the playbook, roles and tasks
• [ ] Verify that everything still works
• [x] Activate conda ewatercycle env on login for shell users

[Peter9192]

@sverhoeven this is as far as I got before my holidays. Unfortunately I didn't have time to do a clean installation of the VM and then test whether the upgrade works with this playbook. My current testing VM might be a bit polluted.

• I think there might still be remnants of old kernels when you do an upgrade from the current environment. I left some commented lines that might be helpful to remove them.
• I didn't remove the sample notebooks, but that seems a reasonable thing to do (maybe in another PR?)
• The grpc4bmi task in jupyter role can also be cleaned further. Since the installation of grpc4bmi itself has moved to ewatercycle role, it's just a collection of docker pulls and singularity builds. More like a 'data' task than 'grpc4bmi'.
• I now run conda init for all users, but ideally the ewatercycle environment should also be activated on login.

[sverhoeven]

Also installed hymuse in conda, but running https://github.com/eWaterCycle/HyMUSE/blob/master/examples/heat/heat_example.py cause 2 python process using 100% cpu and no progress so had to cancel it.

[sverhoeven]

This PR also fixes #50

Final steps before marking ready for review

1. [x] Create vagrant box provisioned from master branch
2. [x] Run ansible with this branch
3. [x] Verify ewatercycle conda env is only kernel
4. [x] Verify ewatercycle conda env is active in Jupyter terminal and after ssh login
5. [x] run https://github.com/eWaterCycle/notebooks/blob/master/bmi/Walrus.ipynb
6. [x] run geoviews example notebook (https://github.com/holoviz/hvplot/blob/master/examples/user_guide/Geographic_Data.ipynb)
7. [x] test code formatter extension
8. [x] run pymt example notebook (sample-notebooks/pymt/pymt_hydrotrend.ipynb)
9. [x] test dask extension
10. [x] Test esmvaltool --help

[sverhoeven]

For any reviewer I would suggest to use the same steps at https://github.com/eWaterCycle/infra/pull/55#issuecomment-659312725 to test if everything works.

[sverhoeven]

During provisioning a vagrant machine from nothing to this branch I noticed some issues. These could also popup during review or deployment to production. I fixed them as I went through the provisioning from scratch.

[nielsdrost]

As a next step, I build the system on the master branch (with the exception of the culler), switched to the "47" branch, and ran ansible. It seems to be unhappy:

TASK [jupyter : grpc4bmi Docker images] ****************************************************************
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/walrus-grpc4bmi', 'singularity': 'ewatercycle-walrus-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/walrus-grpc4bmi", "singularity": "ewatercycle-walrus-grpc4bmi.sif"}, "msg": "Cannot have both the docker-py and docker python modules (old and new version of Docker SDK for Python) installed together as they use the same namespace and cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module (for vagrant.localhost's Python /usr/bin/python3). It is recommended to install the docker module if no support for Python 2.6 is required. Please note that simply uninstalling one of the modules can leave the other module in a broken state."}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/pcrg-grpc4bmi', 'singularity': 'ewatercycle-pcrg-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/pcrg-grpc4bmi", "singularity": "ewatercycle-pcrg-grpc4bmi.sif"}, "msg": "Cannot have both the docker-py and docker python modules (old and new version of Docker SDK for Python) installed together as they use the same namespace and cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module (for vagrant.localhost's Python /usr/bin/python3). It is recommended to install the docker module if no support for Python 2.6 is required. Please note that simply uninstalling one of the modules can leave the other module in a broken state."}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/wflow-grpc4bmi', 'singularity': 'ewatercycle-wflow-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/wflow-grpc4bmi", "singularity": "ewatercycle-wflow-grpc4bmi.sif"}, "msg": "Cannot have both the docker-py and docker python modules (old and new version of Docker SDK for Python) installed together as they use the same namespace and cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module (for vagrant.localhost's Python /usr/bin/python3). It is recommended to install the docker module if no support for Python 2.6 is required. Please note that simply uninstalling one of the modules can leave the other module in a broken state."}
failed: [vagrant.localhost] (item={'docker': 'sverhoeven/marrmot', 'singularity': 'marrmot.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "sverhoeven/marrmot", "singularity": "marrmot.sif"}, "msg": "Cannot have both the docker-py and docker python modules (old and new version of Docker SDK for Python) installed together as they use the same namespace and cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module (for vagrant.localhost's Python /usr/bin/python3). It is recommended to install the docker module if no support for Python 2.6 is required. Please note that simply uninstalling one of the modules can leave the other module in a broken state."}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/hype-grpc4bmi', 'singularity': 'ewatercycle-hype-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/hype-grpc4bmi", "singularity": "ewatercycle-hype-grpc4bmi.sif"}, "msg": "Cannot have both the docker-py and docker python modules (old and new version of Docker SDK for Python) installed together as they use the same namespace and cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module (for vagrant.localhost's Python /usr/bin/python3). It is recommended to install the docker module if no support for Python 2.6 is required. Please note that simply uninstalling one of the modules can leave the other module in a broken state."}

Should I manually un-install it?

[sverhoeven]

The previous task should have installed it, but it was in wrong Python and old package, corrected it in commits. Could you try again?

[nielsdrost]

The previous task should have installed it, but it was in wrong Python and old package, corrected it in commits. Could you try again?

I tried again, new error :-)

TASK [jupyter : Docker Python package] *****************************************************************
ok: [vagrant.localhost]

TASK [jupyter : Remove old Docker-py python2 package] **************************************************
ok: [vagrant.localhost]

TASK [jupyter : Remove old Docker-py python3 package] **************************************************
ok: [vagrant.localhost]

TASK [jupyter : grpc4bmi Docker images] ****************************************************************
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/walrus-grpc4bmi', 'singularity': 'ewatercycle-walrus-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/walrus-grpc4bmi", "singularity": "ewatercycle-walrus-grpc4bmi.sif"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on vagrant.localhost's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: cannot import name '__version__'"}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/pcrg-grpc4bmi', 'singularity': 'ewatercycle-pcrg-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/pcrg-grpc4bmi", "singularity": "ewatercycle-pcrg-grpc4bmi.sif"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on vagrant.localhost's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: cannot import name '__version__'"}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/wflow-grpc4bmi', 'singularity': 'ewatercycle-wflow-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/wflow-grpc4bmi", "singularity": "ewatercycle-wflow-grpc4bmi.sif"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on vagrant.localhost's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: cannot import name '__version__'"}
failed: [vagrant.localhost] (item={'docker': 'sverhoeven/marrmot', 'singularity': 'marrmot.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "sverhoeven/marrmot", "singularity": "marrmot.sif"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on vagrant.localhost's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: cannot import name '__version__'"}
failed: [vagrant.localhost] (item={'docker': 'ewatercycle/hype-grpc4bmi', 'singularity': 'ewatercycle-hype-grpc4bmi.sif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"docker": "ewatercycle/hype-grpc4bmi", "singularity": "ewatercycle-hype-grpc4bmi.sif"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on vagrant.localhost's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.6). The error was: cannot import name '__version__'"}

[SarahAlidoost]

For Walrun notebook FileNotFoundError: [Errno 2] No such file or directory: 'svn': 'svn'

[sverhoeven]

Successfully provisioned server, all checks pass

1. [x] Verify ewatercycle conda env is only kernel, after restart of notebook server
2. [x] Verify ewatercycle conda env is active in Jupyter terminal and after ssh login
3. [x] run https://github.com/eWaterCycle/notebooks/blob/master/bmi/Walrus.ipynb, used ewatercycle/walrus-grpc4bmi:v0.2.0 Docker image and added only_use_cftime_datetimes=False to num2date()
4. [x] run geoviews example notebook (https://github.com/holoviz/hvplot/blob/master/examples/user_guide/Geographic_Data.ipynb), after running bokeh.sampledata.download()
5. [x] test code formatter extension
6. [x] run pymt example notebook (sample-notebooks/pymt/pymt_hydrotrend.ipynb)
7. [x] test dask extension
8. [x] Test esmvaltool --help

[Peter9192]

Wow! Great job Stefan, still so much more for me to learn about this