matplotlib plots not showing when using virtual environment registered as a kernel

[drivenbyentropy]

I apologize if this is not the correct place to report this issue. Please feel free to close the ticket should this be the case.

When creating a new sos notebook in jupyterlab and selecting a python kernel corresponding to a virtual python environment (e.g. created via venv or conda), any plot created with the matplotlib backend fails to show and instead only prints the resulting object (e.g. <Figure size 640x480 with 1 Axes> ).

However, when running the same code using the SoS kernel, the figure is displayed correctly. Is there anything I might be missing when trying to display these plots from a kernel created from a virtual environment?

Please let me know if I can provide any additional information.

Steps to reproduce (assuming jupyterlab and jupyterlab-sos are installed):

• Create a python evironment and install numpy and matplotlib
• Activate environment and register it as a jupyter kernel (python -m ipykernel install --user --name=test)
• Create an SOS notebook and run the code below with the SOS kernel --> plot shows
• Run the same code with the test environment --> plot does not show

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots()
ax.plot(np.arange(0.0, 2.0, 0.01), 1 + np.sin(2 * np.pi * np.arange(0.0, 2.0, 0.01)))

plt.show()

Environment:

>jupyter --version
Selected Jupyter core packages...
IPython          : 7.27.0
ipykernel        : 6.4.1
ipywidgets       : not installed
jupyter_client   : 7.0.3
jupyter_core     : 4.8.1
jupyter_server   : 1.11.0
jupyterlab       : 3.1.12
nbclient         : 0.5.4
nbconvert        : 6.1.0
nbformat         : 5.1.3
notebook         : 6.4.4
qtconsole        : not installed
traitlets        : 5.1.0

>jupyter labextension list
JupyterLab v3.1.12
[SNIP]/.local/share/jupyter/labextensions
        jupyter-matplotlib v0.9.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)

[SNIP]/share/jupyter/labextensions
        jupyterlab-sos v0.8.1 enabled OK (python, jupyterlab-sos)
        transient-display-data v0.4.2 enabled OK (python, transient-display-data)

>python --version
Python 3.9.5

[BoPeng]

I cannot seem to be able to reproduce the problem on my jupyterhub server.

Can you try to use a magic

%matplotlib inline

and let me know if it does anything?

[drivenbyentropy]

@BoPeng thank you for getting back to me so quickly.

I have tired the inline and widget magic and both do not produce the plot in my case. Is there any way I could start debugging this to get a general sense of what the culprit could be?

Just to be certain - was the Python3 kernel in your screenshot created as a virtual environment such as described here (I followed the installation using conda)?

[BoPeng]

That is a large jupyterhub installation and could be different. Let me try it on my local env.

[drivenbyentropy]

Thank you.

Here is the shortest step by step instructions to reproduce the issue on my side:

1) Install miniconda:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
mkdir -p /data/$USER/temp
TMPDIR=/data/$USER/temp bash Miniconda3-latest-Linux-x86_64.sh -p /data/$USER/conda -b

2) Activate base environment and install jupyterlab and jupterlab-sos

source /data/$USER/conda/etc/profile.d/conda.sh
conda activate base
conda install -c conda-forge jupyterlab jupyterlab-sos matplotlib

3) Create virtual envrionment and register as kernel

conda create --name testenv
conda activate testenv
conda install ipykernel matplotlib
python -m ipykernel install --user --name=testenv
conda deactivate

4) Start jupyterlab and test matplotlib. It should work as expected when using the kernel from the base installation but not when using testenv

[BoPeng]

This is what I have

(base) ➜  ~ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
qtconsole        : 4.7.5
ipython          : 7.16.1
ipykernel        : 5.3.2
jupyter client   : 6.1.6
jupyter lab      : 3.1.0
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.3
(base) ➜  ~ jupyter labextension list
JupyterLab v3.1.0
Other labextensions (built into JupyterLab)
   app dir: /Users/bpeng/anaconda3/share/jupyter/lab
        jupyterlab-sos v0.8.1 enabled OK*
        transient-display-data v0.4.1 enabled OK*

   local extensions:
        jupyterlab-sos: /Users/bpeng/vatlab/jupyterlab-sos
        transient-display-data: /Users/bpeng/vatlab/transient-display-data

Build recommended, please run `jupyter lab build`:
    transient-display-data needs to be included in build
    jupyterlab-sos needs to be included in build

I notice that you have a matplotlib extension for jupyterlab, could that be the problem?

[BoPeng]

Works fine on my standalone jupyterlab installation as well. Will try to reproduce it following your instructions tomorrow.

[BoPeng]

I got a more serious error:

jupyter --version
jupyter core     : 4.7.1
jupyter-notebook : 6.4.4
qtconsole        : not installed
ipython          : 7.26.0
ipykernel        : 6.2.0
jupyter client   : 7.0.1
jupyter lab      : 3.1.12
nbconvert        : 6.1.0
ipywidgets       : not installed
nbformat         : 5.1.3
traitlets        : 5.0.5

jupyter labextension list
JupyterLab v3.1.12
/Users/bpeng/miniconda3/envs/testenv/share/jupyter/labextensions
        jupyterlab-sos v0.8.1 enabled OK (python, jupyterlab-sos)
        transient-display-data v0.4.2 enabled OK (python, transient-display-data)

[BoPeng]

Upgrade existing installation to the following versions and cannot reproduce the problem with the new installation

(testsos) ➜  ~ jupyter --version
jupyter core     : 4.7.1
jupyter-notebook : 6.4.4
qtconsole        : 5.1.1
ipython          : 7.27.0
ipykernel        : 6.4.1
jupyter client   : 7.0.3
jupyter lab      : 3.1.12
nbconvert        : 6.1.0
ipywidgets       : 7.6.5
nbformat         : 5.1.3
traitlets        : 5.0.5

> pip list | grep sos
sos                           0.22.5              /Users/bpeng/anaconda3/envs/testsos/lib/python3.8/site-packages
sos-bash                      0.20.0
sos-notebook                  0.22.6              /Users/bpeng/anaconda3/envs/testsos/lib/python3.8/site-packages

[drivenbyentropy]

Thank you again for helping me figuring this out, I got it working now.

The issue was caused by some residual jupyter files from a previous installation that were not correctly deleted in my ~/.local folder prior to switching to conda. After removing these, and starting with a fresh conda installation, I was able to reproduce your "more serious error" you detailed above. Downgrading jupyter client to version 6.1.12 solved this issue along with the installation of qtconsole and ipywidgets.

I appreciate your help. Thanks!

[bioworkflows]

Thanks. I am working on the "more serious bug" (#59) problem and also found the downgrade solution, but I will update sos-notebook to use the new async API and make a release soon(ish).