Jupyter labextension does not find extension from kernel's conda env

StefanBrand commented 3 years ago

Description

I install a JupyterLab extension via mamba in an environment other than the one where JupyterLab is installed. Then I install this environment as a new kernel. The extensions installed in this environment cannot be used in the notebook's kernel.

Reproduce

docker run -p 8888:8888 jupyter/base-notebook jupyter lab
docker exec -it <container-id> bash
mamba install -c conda-forge jupyterlab_widgets (base environment)
conda create -n jlab-test --override-channels --strict-channel-priority -c conda-forge -c nodefaults ipyleaflet
conda activate jlab-test (new environment)
python -m ipykernel install --user --name "jlab-test-kernel" --display-name "JLab Test Kernel (Python)"

Open a new notebook in JupyterLab and execute this snippet (results in Error displaying widget: model not found)

from ipyleaflet import Map, basemaps, basemap_to_tiles, Rectangle
watercolor = basemap_to_tiles(basemaps.Stamen.Watercolor)
m = Map(layers=(watercolor, ), center=(53, 354), zoom=5)
rectangle = Rectangle(bounds=((52, 354), (53, 360)))
m.add_layer(rectangle)
m

cp -r /opt/conda/envs/jlab-test/share/jupyter/labextensions/jupyter-leaflet/ /opt/conda/share/jupyter/labextensions/jupyter-leaflet/
Refresh browser
Run snippet from (7.) again. Map is displayed:

Expected behavior

JupyterLab should detect lab extensions that are installed in the environment used by the notebook's kernel.

Context

Docker image: docker pull jupyter/base-notebook:d75d70f3e661

Troubleshoot Output

$PATH:
    /opt/conda/envs/jlab-test/bin
    /opt/conda/condabin
    /opt/conda/bin
    /usr/local/sbin
    /usr/local/bin
    /usr/sbin
    /usr/bin
    /sbin
    /bin

sys.path:
    /opt/conda/envs/jlab-test/bin
    /opt/conda/envs/jlab-test/lib/python39.zip
    /opt/conda/envs/jlab-test/lib/python3.9
    /opt/conda/envs/jlab-test/lib/python3.9/lib-dynload
    /opt/conda/envs/jlab-test/lib/python3.9/site-packages

sys.executable:
    /opt/conda/envs/jlab-test/bin/python

sys.version:
    3.9.5 | packaged by conda-forge | (default, Jun 19 2021, 00:32:32) 
    [GCC 9.3.0]

platform.platform():
    Linux-5.8.0-59-generic-x86_64-with-glibc2.31

which -a jupyter:
    /opt/conda/envs/jlab-test/bin/jupyter
    /opt/conda/bin/jupyter

pip list:
    Package                       Version
    ----------------------------- -------------------
    argon2-cffi                   20.1.0
    async-generator               1.10
    attrs                         21.2.0
    backcall                      0.2.0
    backports.functools-lru-cache 1.6.4
    bleach                        3.3.0
    branca                        0.4.2
    certifi                       2021.5.30
    cffi                          1.14.5
    decorator                     5.0.9
    defusedxml                    0.7.1
    entrypoints                   0.3
    importlib-metadata            4.6.0
    ipykernel                     5.5.5
    ipyleaflet                    0.14.0
    ipython                       7.25.0
    ipython-genutils              0.2.0
    ipywidgets                    7.6.3
    jedi                          0.18.0
    Jinja2                        3.0.1
    jsonschema                    3.2.0
    jupyter-client                6.1.12
    jupyter-core                  4.7.1
    jupyterlab-pygments           0.1.2
    jupyterlab-widgets            1.0.0
    MarkupSafe                    2.0.1
    matplotlib-inline             0.1.2
    mistune                       0.8.4
    nbclient                      0.5.3
    nbconvert                     6.1.0
    nbformat                      5.1.3
    nest-asyncio                  1.5.1
    notebook                      6.4.0
    numpy                         1.21.0
    packaging                     20.9
    pandocfilters                 1.4.2
    parso                         0.8.2
    pexpect                       4.8.0
    pickleshare                   0.7.5
    pip                           21.1.3
    prometheus-client             0.11.0
    prompt-toolkit                3.0.19
    ptyprocess                    0.7.0
    pycparser                     2.20
    Pygments                      2.9.0
    pyparsing                     2.4.7
    pyrsistent                    0.17.3
    python-dateutil               2.8.1
    pyzmq                         22.1.0
    Send2Trash                    1.7.1
    setuptools                    49.6.0.post20210108
    Shapely                       1.7.1
    six                           1.16.0
    terminado                     0.10.1
    testpath                      0.5.0
    tornado                       6.1
    traitlets                     5.0.5
    traittypes                    0.2.1
    wcwidth                       0.2.5
    webencodings                  0.5.1
    wheel                         0.36.2
    widgetsnbextension            3.5.1
    zipp                          3.4.1

conda list:
    # packages in environment at /opt/conda/envs/jlab-test:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                 conda_forge    conda-forge
    _openmp_mutex             4.5                       1_gnu    conda-forge
    argon2-cffi               20.1.0           py39h3811e60_2    conda-forge
    async_generator           1.10                       py_0    conda-forge
    attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
    backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
    backports                 1.0                        py_2    conda-forge
    backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
    bleach                    3.3.0              pyh44b312d_0    conda-forge
    branca                    0.4.2              pyhd8ed1ab_0    conda-forge
    ca-certificates           2021.5.30            ha878542_0    conda-forge
    certifi                   2021.5.30        py39hf3d152e_0    conda-forge
    cffi                      1.14.5           py39he32792d_0    conda-forge
    decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
    defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
    entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
    geos                      3.9.1                h9c3ff4c_2    conda-forge
    importlib-metadata        4.6.0            py39hf3d152e_0    conda-forge
    ipykernel                 5.5.5            py39hef51801_0    conda-forge
    ipyleaflet                0.14.0             pyhd8ed1ab_1    conda-forge
    ipython                   7.25.0           py39hef51801_0    conda-forge
    ipython_genutils          0.2.0                      py_1    conda-forge
    ipywidgets                7.6.3              pyhd3deb0d_0    conda-forge
    jedi                      0.18.0           py39hf3d152e_2    conda-forge
    jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
    jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
    jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
    jupyter_core              4.7.1            py39hf3d152e_0    conda-forge
    jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
    jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
    ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
    libblas                   3.9.0                9_openblas    conda-forge
    libcblas                  3.9.0                9_openblas    conda-forge
    libffi                    3.3                  h58526e2_2    conda-forge
    libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
    libgfortran-ng            9.3.0               hff62375_19    conda-forge
    libgfortran5              9.3.0               hff62375_19    conda-forge
    libgomp                   9.3.0               h2828fa1_19    conda-forge
    liblapack                 3.9.0                9_openblas    conda-forge
    libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
    libsodium                 1.0.18               h36c2ea0_1    conda-forge
    libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
    markupsafe                2.0.1            py39h3811e60_0    conda-forge
    matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
    mistune                   0.8.4           py39h3811e60_1004    conda-forge
    nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
    nbconvert                 6.1.0            py39hf3d152e_0    conda-forge
    nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
    ncurses                   6.2                  h58526e2_4    conda-forge
    nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
    notebook                  6.4.0              pyha770c72_0    conda-forge
    numpy                     1.21.0           py39hdbf815f_0    conda-forge
    openssl                   1.1.1k               h7f98852_0    conda-forge
    packaging                 20.9               pyh44b312d_0    conda-forge
    pandoc                    2.14.0.3             h7f98852_0    conda-forge
    pandocfilters             1.4.2                      py_1    conda-forge
    parso                     0.8.2              pyhd8ed1ab_0    conda-forge
    pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
    pickleshare               0.7.5                   py_1003    conda-forge
    pip                       21.1.3             pyhd8ed1ab_0    conda-forge
    prometheus_client         0.11.0             pyhd8ed1ab_0    conda-forge
    prompt-toolkit            3.0.19             pyha770c72_0    conda-forge
    ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
    pycparser                 2.20               pyh9f0ad1d_2    conda-forge
    pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
    pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
    pyrsistent                0.17.3           py39h3811e60_2    conda-forge
    python                    3.9.5           h49503c6_0_cpython    conda-forge
    python-dateutil           2.8.1                      py_0    conda-forge
    python_abi                3.9                      2_cp39    conda-forge
    pyzmq                     22.1.0           py39h37b5a0c_0    conda-forge
    readline                  8.1                  h46c0cb4_0    conda-forge
    send2trash                1.7.1              pyhd8ed1ab_0    conda-forge
    setuptools                49.6.0           py39hf3d152e_3    conda-forge
    shapely                   1.7.1            py39ha61afbd_5    conda-forge
    six                       1.16.0             pyh6c4a22f_0    conda-forge
    sqlite                    3.36.0               h9cd32fc_0    conda-forge
    terminado                 0.10.1           py39hf3d152e_0    conda-forge
    testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
    tk                        8.6.10               h21135ba_1    conda-forge
    tornado                   6.1              py39h3811e60_1    conda-forge
    traitlets                 5.0.5                      py_0    conda-forge
    traittypes                0.2.1              pyh9f0ad1d_2    conda-forge
    tzdata                    2021a                he74cb21_0    conda-forge
    wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
    webencodings              0.5.1                      py_1    conda-forge
    wheel                     0.36.2             pyhd3deb0d_0    conda-forge
    widgetsnbextension        3.5.1            py39hf3d152e_4    conda-forge
    xz                        5.2.5                h516909a_1    conda-forge
    zeromq                    4.3.4                h9c3ff4c_0    conda-forge
    zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
    zlib                      1.2.11            h516909a_1010    conda-forge

conda env:
    name: jlab-test
    channels:
      - conda-forge
    dependencies:
      - _libgcc_mutex=0.1=conda_forge
      - _openmp_mutex=4.5=1_gnu
      - argon2-cffi=20.1.0=py39h3811e60_2
      - async_generator=1.10=py_0
      - attrs=21.2.0=pyhd8ed1ab_0
      - backcall=0.2.0=pyh9f0ad1d_0
      - backports=1.0=py_2
      - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
      - bleach=3.3.0=pyh44b312d_0
      - branca=0.4.2=pyhd8ed1ab_0
      - ca-certificates=2021.5.30=ha878542_0
      - certifi=2021.5.30=py39hf3d152e_0
      - cffi=1.14.5=py39he32792d_0
      - decorator=5.0.9=pyhd8ed1ab_0
      - defusedxml=0.7.1=pyhd8ed1ab_0
      - entrypoints=0.3=pyhd8ed1ab_1003
      - geos=3.9.1=h9c3ff4c_2
      - importlib-metadata=4.6.0=py39hf3d152e_0
      - ipykernel=5.5.5=py39hef51801_0
      - ipyleaflet=0.14.0=pyhd8ed1ab_1
      - ipython=7.25.0=py39hef51801_0
      - ipython_genutils=0.2.0=py_1
      - ipywidgets=7.6.3=pyhd3deb0d_0
      - jedi=0.18.0=py39hf3d152e_2
      - jinja2=3.0.1=pyhd8ed1ab_0
      - jsonschema=3.2.0=pyhd8ed1ab_3
      - jupyter_client=6.1.12=pyhd8ed1ab_0
      - jupyter_core=4.7.1=py39hf3d152e_0
      - jupyterlab_pygments=0.1.2=pyh9f0ad1d_0
      - jupyterlab_widgets=1.0.0=pyhd8ed1ab_1
      - ld_impl_linux-64=2.35.1=hea4e1c9_2
      - libblas=3.9.0=9_openblas
      - libcblas=3.9.0=9_openblas
      - libffi=3.3=h58526e2_2
      - libgcc-ng=9.3.0=h2828fa1_19
      - libgfortran-ng=9.3.0=hff62375_19
      - libgfortran5=9.3.0=hff62375_19
      - libgomp=9.3.0=h2828fa1_19
      - liblapack=3.9.0=9_openblas
      - libopenblas=0.3.15=pthreads_h8fe5266_1
      - libsodium=1.0.18=h36c2ea0_1
      - libstdcxx-ng=9.3.0=h6de172a_19
      - markupsafe=2.0.1=py39h3811e60_0
      - matplotlib-inline=0.1.2=pyhd8ed1ab_2
      - mistune=0.8.4=py39h3811e60_1004
      - nbclient=0.5.3=pyhd8ed1ab_0
      - nbconvert=6.1.0=py39hf3d152e_0
      - nbformat=5.1.3=pyhd8ed1ab_0
      - ncurses=6.2=h58526e2_4
      - nest-asyncio=1.5.1=pyhd8ed1ab_0
      - notebook=6.4.0=pyha770c72_0
      - numpy=1.21.0=py39hdbf815f_0
      - openssl=1.1.1k=h7f98852_0
      - packaging=20.9=pyh44b312d_0
      - pandoc=2.14.0.3=h7f98852_0
      - pandocfilters=1.4.2=py_1
      - parso=0.8.2=pyhd8ed1ab_0
      - pexpect=4.8.0=pyh9f0ad1d_2
      - pickleshare=0.7.5=py_1003
      - pip=21.1.3=pyhd8ed1ab_0
      - prometheus_client=0.11.0=pyhd8ed1ab_0
      - prompt-toolkit=3.0.19=pyha770c72_0
      - ptyprocess=0.7.0=pyhd3deb0d_0
      - pycparser=2.20=pyh9f0ad1d_2
      - pygments=2.9.0=pyhd8ed1ab_0
      - pyparsing=2.4.7=pyh9f0ad1d_0
      - pyrsistent=0.17.3=py39h3811e60_2
      - python=3.9.5=h49503c6_0_cpython
      - python-dateutil=2.8.1=py_0
      - python_abi=3.9=2_cp39
      - pyzmq=22.1.0=py39h37b5a0c_0
      - readline=8.1=h46c0cb4_0
      - send2trash=1.7.1=pyhd8ed1ab_0
      - setuptools=49.6.0=py39hf3d152e_3
      - shapely=1.7.1=py39ha61afbd_5
      - six=1.16.0=pyh6c4a22f_0
      - sqlite=3.36.0=h9cd32fc_0
      - terminado=0.10.1=py39hf3d152e_0
      - testpath=0.5.0=pyhd8ed1ab_0
      - tk=8.6.10=h21135ba_1
      - tornado=6.1=py39h3811e60_1
      - traitlets=5.0.5=py_0
      - traittypes=0.2.1=pyh9f0ad1d_2
      - tzdata=2021a=he74cb21_0
      - wcwidth=0.2.5=pyh9f0ad1d_2
      - webencodings=0.5.1=py_1
      - wheel=0.36.2=pyhd3deb0d_0
      - widgetsnbextension=3.5.1=py39hf3d152e_4
      - xz=5.2.5=h516909a_1
      - zeromq=4.3.4=h9c3ff4c_0
      - zipp=3.4.1=pyhd8ed1ab_0
      - zlib=1.2.11=h516909a_1010
    prefix: /opt/conda/envs/jlab-test

Browser Output

Uncaught (in promise) Error: Module jupyter-leaflet, semver range ^0.14.0 is not registered as a widget module

welcome[bot] commented 3 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

agoose77 commented 3 years ago

Thanks for filing an issue!

As far as JupyterLab is concerned, the kernel is separate from the frontend. This is by design; kernels don't need to be installed into virtual environments (or even conda environments), they just need to have a kernelspec installed into the right location (which you are doing manually).

Because of this separation, it is (in my opinion) unlikely that this will ever be changed to fulfil your use case.

jasongrout commented 3 years ago

Elaborating on this: the frontend and its extensions serve many kernels. If the kernels have different extensions, or conflicting extensions, it's not clear what should be loaded and how to handle conflicts. I agree in general this is unlikely to change in the architecture. Right now, the frontend environment is the one that determines what extensions are loaded.

In your specific case, it sounds like you are talking about ipywidgets extensions. ipywidgets itself has a mechanism for dealing with multiple versions of the same extension, so it may be possible to have something work in this specific case. However, I think it will require a decent amount of work to explore this to even come up with a proposal to discuss, for example:

Getting information from the kernelspec about where to look for ipywidgets extensions
Loading ipywidgets extensions dynamically at runtime (rather than just at startup)
ipywidgets knowing what kernels are loading what extensions, and restricting the kernel to its own extensions instead of other kernels' extensions.

If someone wants to explore putting together a precise proposal to discuss, let us know and we can help you get started. Probably a lot of this code would be implemented in ipywidgets as well.

fcollonval commented 3 years ago

Small note, the configuration parameter extra_labextensions_path could be used to address this (not tested):

https://github.com/jupyterlab/jupyterlab_server/blob/8bcdf52a06bd81e649825e153a3f93947064fe23/jupyterlab_server/config.py#L198-L200

In your case, you could try:

jupyter lab --LabApp.extra_labextensions_path="/opt/conda/envs/jlab-test/share/jupyter/labextensions" --LabApp.extra_labextensions_path="/opt/conda/share/jupyter/labextensions/jupyter-leaflet/"

Note the need for setting two paths is due to that bug: https://github.com/ipython/traitlets/issues/668

StefanBrand commented 3 years ago

@fcollonval Thank you for the command. It actually works for our use case :+1:

StefanBrand commented 3 years ago

I'm now trying to employ the jupyter_lab_config.py to set the extra_labextensions_path. I see multiple configuration options that seem relevant:

## Extra paths to look for federated JupyterLab extensions
#  Default: []
# c.LabServerApp.extra_labextensions_path = []
--
## The standard paths to look in for federated JupyterLab extensions
#  Default: []
# c.LabServerApp.labextensions_path = []
--
## Extra paths to look for federated JupyterLab extensions
#  Default: []
# c.LabApp.extra_labextensions_path = []
--
## The standard paths to look in for federated JupyterLab extensions
#  Default: []
# c.LabApp.labextensions_path = []

I have some questions:

Where to put the jupyter_lab_config.py so that it is applied automatically? The home directory is not an option as it is being mounted into the container in our JupyterHub setup.
What's the difference between the LabApp and LabServerApp options, if any?
How does JupyterLab even pick up any extensions, if labextensions_path is [] by default?

agoose77 commented 3 years ago

Config Locations

If you run jupyter --paths in a terminal, you will be presented with a breakdown of the various locations that Jupyter uses for configuration and data.

You can place a configuration file in any of the config locations and it will be picked up by the configuration system. If you have multiple configuration files, all will be considered, but higher priority configs will override lower priority ones.

`LabApp` vs `LabServerApp`

The inheritance of the configuration objects is roughly

LabApp -> LabServerApp -> ExtensionApp -> JupyterApp

As to what jupyterlab_server does, I'm actually not entirely sure. I think it provides a way to develop with a subset of JupyterLab, but one of the more knowledgeable developers might correct me here.

In general, a lot of the Jupyter stack is built upon traitlets, which provides (amongst other things) the configuration system that we're discussing here. In the traitlets library, class inheritance is used as a way to support configuration inheritance, which is why there are often relatively deep class hierarchies.

Given that LabApp inherits from LabServerApp, I would suggest setting the trait on LabServerApp, so that child classes are also given the same configuration. I doubt that there are any other applications that inherit from LabServerApp right now (at least, in your environment), but that's not important.

How JupyterLab finds extensions

The comment in the configuration file concerning the default labextensions path is a little misleading. traitlets has two mechanisms for providing default values

Trait constructor default values
Class instance default methods

The latter mechanism allows a HasTraits class (i.e. nearly all Python classes in the Jupyter codebase) to define a method that is called when the trait needs a default value. This default doesn't show up in the configuration file comments, but it is used in the absence of a value. The labextensions_path trait is defined in LabConfig, which is inherited from by LabServerApp (which inherits from multiple classes).

The default for this trait is set in this method.

StefanBrand commented 3 years ago

Thank very much you for the comprehensive write-up! :heart:

We ended up copying jupyter_lab_config.py to /etc/jupyter/ within the Dockerfile(s).

The jupyter_lab_config.py would look like this (in the context of this issue and its OP):

# Configuration file for lab.

## Extra paths to look for federated JupyterLab extensions
c.LabServerApp.extra_labextensions_path = ["/opt/conda/envs/jlab-test/share/jupyter/labextensions"]

This is the actual commit that contains the change for our use case: https://github.com/eurodatacube/base-images/pull/4/commits/bbf9bf25e4ee1088eb17940338ca87404418bcdb

fcollonval commented 3 years ago

Thanks @StefanBrand for sharing your findings.

jupyterlab / jupyterlab