Single Conda Jupyter kernel

eWaterCycle / infra

Instructions for system administrators to deploy the eWaterCycle platform

Apache License 2.0

0 stars 0 forks source link

Single Conda Jupyter kernel #47

Closed sverhoeven closed 4 years ago

sverhoeven commented 4 years ago

The packages available in OS python and conda are different. Some packages can only be easily installed in conda like ESMValtool. Picking a kernel for a notebook can be confusing.

sverhoeven commented 4 years ago

During planing we decided to give Jupyter a single kernel using conda.

sverhoeven commented 4 years ago

Kernels used by Jupyter are in /usr/local/share/jupyter/kernels . When installing Jupyter, a kernel using the same Python as Jupyter is added. This kernel can be removed with jupyter kernelspec remove python3.

sverhoeven commented 4 years ago

In JupyterLab environment a terminal can be started and also users can login with ssh. The Python you get in your path should be the Conda Python.

Peter9192 commented 4 years ago

jupyter kernelspec remove -f python3

(-y option didn't work and left ansible hanging) replaces

python3 /usr/local/share/jupyter/kernels/python3 with python3 /usr/local/lib/python3.6/dist-packages/ipykernel/resources i.e. it defaults to system python. Additionally need to set disable the native kernel as explained here and here for jupyterhub instances: c.Spawner.args = ["--KernelSpecManager.ensure_native_kernel=False"]

Peter9192 commented 4 years ago

Might be nice to change the name of the conda kernel. Right now it is 'Python 3 (Conda)'. Just 'Python 3' would be nicer. This name is already taken by the system kernel, but we're removing that anyway. Or maybe we could do 'Python 3 (Default)'

sverhoeven commented 4 years ago

The OS Python is used for many tasks to install dependencies used in a notebook or labextension. We should also install these in the Conda environment.

Some pip tasks can be update by just changing the executable to the conda pip and others might need to be split up in a pip task with OS python and task with Conda Python.

Peter9192 commented 4 years ago

We should also install these

I added a task to the draft PR in #55

Peter9192 commented 4 years ago

Roles in Jupyter playbook that may need to be updated:

[x] Hymuse - uses systems pip3 and python3
[x] ESMValTool - already uses conda and installs all esmvaltool deps in the conda base env (different versions than system pip)
[x] Pymt - already uses conda

And within the jupyter role there are:

[x] Hub - installed using pip3 (may be okay?)
[x] Lab - many pip3 installs with pinned versions - may have discrepancies with current conda versions (determined by esmvaltool env above)
[x] GRPC4BMI - Currently uses systems pip3

So many of the (old) lab environment governed by pip is already present in conda due to ESMValTools dependencies, but we may need different versions and additional packages.

Peter9192 commented 4 years ago

We decided to switch entirely to conda, and create an 'all-encompassing' ewatercycle environment, which will also hold the default jupyterhub/lab installation. It is possible to add dedicated conda envs for special software such as hymuse (if really necessary), or to add kernels for e.g. older versions of the ewatercycle environment later on.