ipympl when jupyter and ipython kernel are in different conda environment

[matmel]

Hi,

Some context

I typically install the jupter notebook in a conda environment, and create a shortcut on the desktop to launch jupyter-notebook.exe (windows 10 64 bits). Then I create an ipython kernel with all dependencies I want to work with in an other environment, and I register this kernel-in-a-conda-environment with the command

ipython kernel install --user --name NameOfMyNewKernel

I find this workflow relatively clean because I can chose when I want to update either my kernel (bleeding edge or very old depending on this situation) independently from the notebook itself.

The problem

As far as I understand ipympl is really 2 packages.

• One is a python package called ipympl that must be installed together with your ipython kernel, because you need to write import ipympl in your notebook.
• The other one is a javascript package (I supposed it is called also ipympl) that must be installed together with the jupyter notebook application. I don't know if the python package is required or not. You also need to make sure that the javascript package is registered as an extension (jupyter nbextension enable --py --sys-prefix ipympl) by jupyter.

If this is documented somewhere it is well hidden ;-)

The workaround

I installed 2 times the ipympl package, one in my kernel environment, the other in the jpuyter environment. For the latter case, because the python package is also shipped in the ipympl conda package, matplotlib is installed but I think that is useless. With this, everyting works, I am able to see interactive plots in my notebook.

Before figuring out that I needed to install ipympl in the jupyter environment, I received weird error messages without really explaining what the problem was.

Proposal

1. Document the problem. I suppose that this repository README coud explain this, but I suppose it should be also documented somewhere in matplotlib and/or jupyter documentation. I must admit I find the jupyter documentation really hard to navigate though.
2. Split ipympl conda package in two: one that contains the python package and the other one containing the javascript package.

Thanks

[SylvainCorlay]

@matmel you have a good understanding of the problem.

In fact, the core ipywidgets package has this division in two different ones:

• widgetsnbextension is the notebook extension including the static assets.
• ipywidgets only holds the python backend.

The dependency of ipywidgets on widgetsnbextension is merely an old convenience function for users familiar with installing only one package.

Unfortunately, we have not performed this kind of separation in the custom widgets (ipympl, pythreejs, bqplot etc...) But it should really be the case.

[mangecoeur]

Just to chime in that splitting the packages would make sense - I think there is still some general confusion in the whole ecosystem because of the transition from ipython being one thing to the current jupyter+kernels, even more so with jupyterlab (which AFAIU is basically a js/nodejs app). But i think the going forward for most people the jupterter(lab) frontend and the kernels will be running in different environments.

[thomasaarholt]

I was just trying out nb_conda_kernels, which lets one install jupyter lab in one environment, and "call" all other environments from there - so one only needs to install jupyterlab once. Problem is that ipympl seems to need to be installed, and ipympl depends on notebook down the chain of dependencies, which in turn installs a large number of dependencies.

I think this is essentially the same thing that this issue is about. The issue seems to be ipympl -> ipywidgets -> widgetsnbextension -> notebook.

Ref: Anaconda-Platform/nb_conda_kernels#201

[wholtz]

This issue is still causing people headaches. Here is Dockerfile that demonstrates the problem under JupyterLab 3:

# this is a micromamba:0.19.1 image, using digest instead of tag for reproducibility
FROM mambaorg/micromamba@sha256:1ab2666fcd1559e9865d599433671f97cb735103af9351e5e10a3f0d5b6048c8

# Create 2 conda environments:
# "lab" environment will run the jupyterlab server
# "kernel" environment will be active in the "CondaKernel" kernel within a jupyter notebook

RUN micromamba create --yes --name lab --channel conda-forge \
       jupyterlab=3.2.6 \
       jupyterlab_widgets=1.0.2 \
       python=3.10.1 \
       xorg-libxrender=0.9.10 && \
    micromamba create --yes --name kernel --channel conda-forge \
       ipympl=0.8.5 \
       ipywidgets=7.6.5 \
       matplotlib=3.5.0 \
       python=3.10.1 && \
    micromamba clean --all --yes

# activate the "kernel" environment to be active within in the next RUN command
ARG MAMBA_DOCKERFILE_ACTIVATE=1 
ENV ENV_NAME=kernel

# create a kernel.json file for using the "kernel" conda environment in a notebook
RUN python -m ipykernel install --user --name conda-kernel --display-name CondaKernel

# activate the "lab" environment for when the CMD executes
ENV ENV_NAME=lab

EXPOSE 8888
WORKDIR /home/micromamba
CMD ["jupyter", "lab", "--ip=0.0.0.0", "--ServerApp.token=''", "--ServerApp.root_dir=/home/micromamba"]

If you then open a notebook using the CondaKernel kernel, you can run:

import matplotlib.pyplot as plt
%matplotlib widget
fig = plt.figure()
plt.plot([1,2,7])

and I get

[<matplotlib.lines.Line2D at 0x7fa035b362f0>]
Error displaying widget: model not found

Our use case is that we have a JupyterHub server used by many people who use custom kernels. Currently ipympl is installed in the server environment, but user who want to make use of ipympl in their custom kernels need to exactly match the version of ipympl in the server environment. This is fragile and not expected by users. We would like to avoid installing ipympl in the server environment so that users can have more freedom to install a version of ipympl that works for them. Splitting of ipympl into two packages like ipywidgets would allow us to only install the javascript part on the server leaving the users to install the other part in their kernels.

[ianhi]

Splitting of ipympl into two packages like ipywidgets would allow us to only install the javascript part on the server leaving the users to install the other part in their kernels.

@wholtz ipympl is also split into two packages. Would https://www.npmjs.com/package/jupyter-matplotlib fix the problem?

[wholtz]

@ianhi what is the proper way to install that npm package so that jupyterlab can find it?

[ianhi]

How do you do it for ipywidgets? I'd try to replicate that. But naively I'd expect: jupyter labextension install jupyter-matplotlib

[wholtz]

Thanks @ianhi - I was able to install the npm with:

jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-matplotlib@0.10.4

And this did give me the desired behavior!

But....

• The documentation around how to use ipympl in a kernel is lacking. I didn't think to try using jupyter labextension ... because all the setup instructions for jupyterlab 3+ only use pip.
• ipywidgets makes this a little nicer by having a python package (jupyterlab-widgets) for installing the javascript part.

[martinRenou]

Ipympl installs the Javascript part automatically (part of the same package, not separated like ipywidgets does).

It is unclear to me why you need to install the labextension manually.

What does jupyter labextension list give you? (in the case you don't install jupyter-matplotlib manually)

[ianhi]

The documentation around how to use ipympl in a kernel is lacking. I didn't think to try using jupyter labextension ... because all the setup instructions for jupyterlab 3+ only use pip.

Agreed that there's no documentation of this. It would be good to add some! I think this never got done as none of the package maintainers have this use case. I at least don't fully understand the constraints of the goals. Which is to say: PRs welcome :)

ipywidgets makes this a little nicer by having a python package (jupyterlab-widgets) for installing the javascript part.

This isn't quite analogous. jupyterlab-widgets provides the managing code that enables widgets of any sort to work in jupyterlab - any widget needs this to work in jupyterlab. The ipywidgets equivalent to jupyter-matplotlib https://www.npmjs.com/package/@jupyter-widgets/controls which is bundled with the installation of pip install ipywidgets. I'm actually surprised that you didn't run into the same issues with ipywidgets (e.g. intslider) as you did with ipympl