Can't get "%matplotlib widget" to work with SoS kernel

[nibheis]

Hi SoS!

We have updated our JupyterHub/JupyterLab installation for the upcoming semester. For ipyml/matplotlib, I followed the instructions there: https://matplotlib.org/ipympl/installing.html

Here is the current versions installed:

python==3.11.5
jupyterlab==3.6.5
matplotlib==3.7.2
matplotlib-inline==0.1.6
transient-display-data==0.4.3
jupyter-contrib-core==0.4.2
ipympl==0.9.3
lab extension: jupyter-matplotlib 0.11.3

And on the SoS side:

jupyterlab-sos==0.9.0
sos==0.24.1
sos-bash==0.21.0
sos-javascript==0.20.0
sos-julia==0.20.0
sos-matlab==0.20.2
sos-notebook==0.24.2
sos-papermill==0.2.1
sos-python==0.20.0
sos-r==0.20.1

Here is my simple test cell:

%matplotlib widget
import numpy
from matplotlib import pyplot
x = numpy.linspace(0 ,10*numpy.pi, 10**3)
y = numpy.sin(x) + numpy.cos(x)
pyplot.title('plot fonction y=cos(x)+sin(x)')
pyplot.plot(x,y)

And here are my 3 different scenarios results:

1. Using the Python3 kernel directly: I get the plot in a widget (resizable, etc.) - it works.
2. Using SoS kernel, and using the Python3 sub-kernel - the plot shows in an inline plot (not resizable)
3. Using SoS kernel only - an empty widget appears

I have tried downgrading ipympl/matplotlib - but it stays that way. Do you see any other components that I should try upgrading/downgrading? Is JupyterLab 3.6.x supported by SoS? I don't expect much API changes over the 3.4.x series... but you know better.

[nibheis]

Hi SoS! The semester is about to start and we have one course blocked by this annoying problem. We can not consider downgrading our JupyterHub/JupyterLab installation for a single course, but we certainly cannot leave this course alone without trying to fix this problem.

Can anyone confirm that this ticket is opened in the right place? (SoS components are spread of several github repos) Can anyone help us and point us to where we can dig and investigate? Any help is welcome !

[BoPeng]

ok, I can reproduce your problem.

The plot in the Python3 subkernel does not work, which is more or less expected because the sos-kernel sits in between and does not know how to handle the communications from the frontend javascript.

The plot in the SoS kernel starts because sos renders the output, but it does not know how to respond to frontend communications from the jupyter-matplotlib library. I am having a look but I doubt that there can be a quick fix for this.

[BoPeng]

It is a compatibility issues with the jupyterlab-sos extension with the jupyter-matlablib extension. It is not as hopeless as I had expected (if jupyterlab-sos needs to mess with the communications of jupyter-matlablib) but it is not something that I can address quickly.

[nibheis]

Hi @BoPeng !

Thank you so much for having a look. Yes, matplotlib 3.5.3 widgets used to work with jupyerlab-sos 0.8.8 and jupyterlab 3.4.7, using the SoS kernel directly for Python cells.

Can I help with something on this?

[BoPeng]

Not really, but I will investigate further and hopefully get it fixed this weekend.

[BoPeng]

Unfortunately, the following combination does not work so I am not able to figure out what has been changed.

matplotlib 3.5.3 jupyerlab-sos 0.8.8 jupyterlab 3.4.7 ipympl 0.9.3

[nibheis]

Hi @BoPeng ,

Our last known working SoS / matplotlib setup was:

python 3.8.x

jupyterlab 3.4.7
jupyterlab-sos 0.8.8
ipympl 0.8.4
matplotlib 3.5.3
matplotlib-inline 0.1.6

jupyterlab-extension: jupyter-matplotlib 0.10.4

sos 0.23.4
sos-bash 0.20.0
sos-javascript 0.18.2
sos-julia 0.18.5
sos-matlab 0.18.6
sos-notebook 0.23.4
sos-papermill 0.2.1
sos-python 0.18.4
sos-r 0.19.6

[nibheis]

Please also check here:

https://matplotlib.org/ipympl/installing.html#compatibility-table

for the compatible versions of ipympl/jupyter-matplotlib/JupyterLab/Matplotlib.

[BoPeng]

ok, I created a separate condo environment with the exact version of python and relevant packages, and tried with a separate machine/browser, and could not make it work. This means that somethings else was involved and I have to dig further into the root of the problem instead of simply comparing versions.

I note that ipympl works for JLab 4.x so I will work directly on the latest version of jupyterlab-sos instead of the version for JLab 3.x. I will back port the fix if needed though.

Technical notes:

1. ipywidgets saves model_id, which is the id of a comm

https://github.com/jupyter-widgets/ipywidgets/blob/52663ac472c38ba12575dfb4979fa2d250e79bc3/packages/base-manager/src/manager-base.ts#L359

2. It then tries to connect to the widget with the customized comm

https://github.com/jupyter-widgets/ipywidgets/blob/52663ac472c38ba12575dfb4979fa2d250e79bc3/packages/base-manager/src/manager-base.ts#L218

but the comm is already disconnected or destroyed. It looks like the sos kernelsomehow killed thecomm(although technically speaking it is not aware of the existence of thecomm`.

[BoPeng]

It was due to an incompatible change of ipykernel, downgrading ipykernel to <=6.18 should solve the problem but I am submitting a patch to handle the most recent version of ipykernel.

[BoPeng]

I have fixed the bug and released sos-notebook 0.24.4. This should work for JupyterLab 3.x and 4.x and any version of jupyterlab-sos.

Please let me know if the new version fixes the problem. It is possible to enable widget for subkernels but that has to be a separate ticket.

[nibheis]

Hi @BoPeng,

I just ran a basic test, and it looks like it is working. Thank you so much.

Here are my 3 different scenarios results:

1. Using the Python3 kernel directly: I get the plot in a widget (resizable, etc.) - it works.
2. Using SoS kernel, and using the Python3 sub-kernel - the plot shows in an inline plot (not resizable)
3. Using SoS kernel only: I get the plot in a widget (resizable, etc.) - it works too.

I cannot deploy on production servers right now because we're in the middle of a large class and I would prefer my colleagues to confirm that it also works for them.

So, I'll ask my colleagues to confirm it works on the test platform, then I'll close this ticket. Thank you again !

[nibheis]

I confirm that widgets now work with sos-notebook==0.24.4, from the SoS kernel. Your help was much appreciated, thanks again. This fix is enough for us, as we recovered previous level of functionality, so I'll close this ticket. Looking forward to testing widgets in subkernels - I will be following ticket #350.

[BoPeng]

Scenario 2 actually works, see https://github.com/vatlab/sos-notebook/issues/350#issuecomment-1738524482 .