Revise `pyzmq` requirements in documentation

[miloaissatu]

Having kernel crashes on on python3.11.3, ipython8.12.0, notebook6.5.4, jupyterlab 3.6.3 when running a shell/magic command after adding anything other than 'ignore' (without additional specific filters).

Notebook with reproduction examples and environment info: filterwarnings_kernelcrash.zip

Can't replicate this behaviour on python3.10 using docker-stacks minimal notebook.

Refer to attached ipynb in zip for abit more info, but brief version below.

Example cell that results in immediate crash

import warnings
warnings.filterwarnings('once')
!df -h

Example cells that don't result in immediate kernel crash

##### Cell 1
import warnings
warnings.filterwarnings('once')

##### Cell 2
warnings.filters

###### Cell 3
!df -h

Environment info, reduced to packages I felt were relevant, refer to zipped notebook for full list. Jupyter installed via pip install jupyterhub jupyterlab notebook so just latest stable picked up from pypi.

!cat /etc/redhat-release
Rocky Linux release 8.7 (Green Obsidian)

!uname -r
4.18.0-372.9.1.el8.x86_64

%pip freeze
ipykernel==6.22.0
ipympl==0.9.3
ipython==8.12.0
ipython-genutils==0.2.0
ipywidgets==8.0.6
jupyter-events==0.6.3
jupyter-resource-usage==0.7.2
jupyter-server-mathjax==0.2.6
jupyter-telemetry==0.1.0
jupyter-ydoc==0.2.4
jupyter_client==8.2.0
jupyter_core==5.3.0
jupyter_server==2.5.0
jupyter_server_fileid==0.9.0
jupyter_server_terminals==0.4.4
jupyter_server_ydoc==0.8.0
jupyterhub==3.1.1
jupyterlab==3.6.3
jupyterlab-code-formatter==1.6.1
jupyterlab-favorites==3.1.1
jupyterlab-pygments==0.2.2
jupyterlab-system-monitor==0.8.0
jupyterlab-topbar==0.6.1
jupyterlab-widgets==3.0.7
jupyterlab_server==2.22.1
nbclassic==0.5.5
nbclient==0.7.3
nbconvert==7.3.1
nbdime==3.1.1
nbformat==5.8.0
notebook==6.5.4
notebook_shim==0.2.2
pyzmq==25.0.2

[JasonWeill]

I tried opening the notebook in JupyterLab Desktop (3.6.1-1) and did not see the crash mentioned. Might be specific to Notebook 6, not Lab.

[JasonWeill]

Do you see any errors in the terminal where you run Jupyter Notebook at the time of your crash? If so, can you include these in this issue?

[RRosio]

Just wanted to mention that I tried reproducing this with Notebook 6.5.4 and Python3.11.3 but was not able to reproduce the crash.

[miloaissatu]

Do you see any errors in the terminal where you run Jupyter Notebook at the time of your crash? If so, can you include these in this issue?

Nothing visible in the notebook, I can show what's written on logs by the notebook/lab server but it doesn't give much beyond kernel restarted.

Is there another way to get more relevant logs?

[miloaissatu]

I tried opening the notebook in JupyterLab Desktop (3.6.1-1) and did not see the crash mentioned. Might be specific to Notebook 6, not Lab.

I had similar behaviour on lab and notebook. What ipython/ipykernel version do you have?

[miloaissatu]

Dockerfile.txt Dockerfile building FROM dockerhub python:3.11.3 jupyterlab interface -> new notebook -> run import warnings; warnings.filterwarnings('once'); !df -h = crash jupyter lab -> classic notebook interface -> new notebook -> run import warnings; warnings.filterwarnings('once'); !df -h = crash instead of starting via jupyterlab, start jupyter notebook -> run import warnings; warnings.filterwarnings('once'); !df -h = crash

[miloaissatu]

Do you see any errors in the terminal where you run Jupyter Notebook at the time of your crash? If so, can you include these in this issue?

Nothing apart from the kernel restarting, let me know if there's any flags or vars to set so that it might give more info.

jovyan@522df0638418:~$ jupyter notebook
[I 05:05:33.233 NotebookApp] Writing notebook server cookie secret to /juno-tmp/jupyter-data/runtime/notebook_cookie_secret

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[W 05:05:33.723 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[C 05:05:33.723 NotebookApp] You must use Jupyter Server v1 to load JupyterLab as notebook extension. You have v2.5.0 installed.
    You can fix this by executing:
        pip install -U "jupyter-server<2.0.0"
[I 05:05:33.724 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 05:05:33.724 NotebookApp] Jupyter Notebook 6.5.4 is running at:
[I 05:05:33.724 NotebookApp] http://522df0638418:8888/?token=3926d47369eab6e1cba4560c2f4fb78eddac15078a918dd7
[I 05:05:33.724 NotebookApp]  or http://127.0.0.1:8888/?token=3926d47369eab6e1cba4560c2f4fb78eddac15078a918dd7
[I 05:05:33.724 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 05:05:33.727 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///juno-tmp/jupyter-data/runtime/nbserver-10-open.html
    Or copy and paste one of these URLs:
        http://522df0638418:8888/?token=3926d47369eab6e1cba4560c2f4fb78eddac15078a918dd7
     or http://127.0.0.1:8888/?token=3926d47369eab6e1cba4560c2f4fb78eddac15078a918dd7
[I 05:05:36.578 NotebookApp] 302 GET /?token=3926d47369eab6e1cba4560c2f4fb78eddac15078a918dd7 (192.168.200.10) 0.620000ms
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[W 05:05:40.109 NotebookApp] 404 GET /api/events/subscribe?token=ddda4f4b5c17fe30b254ff76cc250e5b113cc11e3f3f7875 (192.168.200.10) 7.970000ms referer=None
[I 05:05:42.326 NotebookApp] Writing notebook-signing key to /juno-tmp/jupyter-data/notebook_secret
[W 05:05:42.327 NotebookApp] Notebook crash.ipynb is not trusted
[W 05:05:42.332 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20230426050533 (192.168.200.10) 1.790000ms referer=http://127.0.0.1:8888/notebooks/crash.ipynb
[I 05:05:42.397 NotebookApp] Kernel started: aa6668bd-3833-49e4-a4dd-71d5f2a5eb95, name: python3
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 05:05:48.388 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
WARNING:root:kernel aa6668bd-3833-49e4-a4dd-71d5f2a5eb95 restarted
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

[miloaissatu]

did a more vanilla approach and had the same crash Dockerfile

FROM python:3.11.3-bullseye
EXPOSE 8888

then

docker build -t simple .
docker run --rm -it --ip=0.0.0.0 -p 8888:8888 simple bash

in container bash shell

python3 -m pip install jupyter
jupyter notebook --allow-root --ip=0.0.0.0 --port=8888

clicking on the notebook url to open in browser, new notebook, import warnings; warnings.filterwarnings('once'); !df -h crashes the kernel

This is whats printed to screen on the bash shell where I started jupyter notebook

root@0051e2299726:/# jupyter notebook --allow-root --ip=0.0.0.0 --port=8888

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[W 06:08:53.667 NotebookApp] Loading JupyterLab as a classic notebook (v6) extension.
[C 06:08:53.667 NotebookApp] You must use Jupyter Server v1 to load JupyterLab as notebook extension. You have v2.5.0 installed.
    You can fix this by executing:
        pip install -U "jupyter-server<2.0.0"
[I 06:08:53.668 NotebookApp] Serving notebooks from local directory: /
[I 06:08:53.668 NotebookApp] Jupyter Notebook 6.5.4 is running at:
[I 06:08:53.668 NotebookApp] http://0051e2299726:8888/?token=0f0d257a70bace366e4f5884155b3b4a2e3cf27b7ca7843c
[I 06:08:53.668 NotebookApp]  or http://127.0.0.1:8888/?token=0f0d257a70bace366e4f5884155b3b4a2e3cf27b7ca7843c
[I 06:08:53.668 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 06:08:53.671 NotebookApp] No web browser found: could not locate runnable browser.
[C 06:08:53.671 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-70-open.html
    Or copy and paste one of these URLs:
        http://0051e2299726:8888/?token=0f0d257a70bace366e4f5884155b3b4a2e3cf27b7ca7843c
     or http://127.0.0.1:8888/?token=0f0d257a70bace366e4f5884155b3b4a2e3cf27b7ca7843c
[I 06:08:56.080 NotebookApp] 302 GET /?token=0f0d257a70bace366e4f5884155b3b4a2e3cf27b7ca7843c (192.168.200.10) 0.930000ms
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 06:09:04.306 NotebookApp] Creating new notebook in
[I 06:09:04.355 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[I 06:09:05.128 NotebookApp] Kernel started: 92c040da-dfef-49c5-87eb-ef8b4e721c08, name: python3
[W 06:09:05.140 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20230426060853 (192.168.200.10) 8.430000ms referer=http://127.0.0.1:8888/notebooks/Untitled.ipynb?kernel_name=python3
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 06:09:32.117 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel 92c040da-dfef-49c5-87eb-ef8b4e721c08 restarted
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[I 06:11:06.084 NotebookApp] Saving file at /Untitled.ipynb
/usr/local/lib/python3.11/site-packages/nbformat/__init__.py:129: MissingIDFieldWarning: Code cell is missing an id field, this will become a hard error in future nbformat versions. You may want to use `normalize()` on your notebooks before validations (available since nbformat 5.1.4). Previous versions of nbformat are fixing this issue transparently, and will stop doing so in the future.
  validate(nb)
/usr/local/lib/python3.11/site-packages/notebook/services/contents/manager.py:353: MissingIDFieldWarning: Code cell is missing an id field, this will become a hard error in future nbformat versions. You may want to use `normalize()` on your notebooks before validations (available since nbformat 5.1.4). Previous versions of nbformat are fixing this issue transparently, and will stop doing so in the future.
  validate_nb(model['content'])

[RRosio]

Hi @miloaissatu, thanks for the information. When you encounter this error, are you using one of the official docker stack images from https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html?

[miloaissatu]

I didn't use the official docker stacks images as I didn't see a python3.11 version available.

[miloaissatu]

@RRosio ### & @JasonWeill , were you using conda/mamba/micromamba to install jupyter components and dependencies when attempting to reproduce the crash?

i've been able to consistently reproduce this behaviour on python3.8,3.9,3.10,3.11 on rockylinux, debian, ubuntu with python installed via their package managers and also from source. On some of these versions the kernel crash wasn't instant but would happen after cycling through a few cells using shell/magic functions e.g. !df -h, !ps -ef, %pip show notebook, %pip freeze. I don't get a crash on an old centos7 container i've got where jupyter was installed on python3.8 2 years ago (2 year old versions of lab/notebook/ipython/ipykernel etc)

the only time i had success with not being able to intentionally crash the kernel on the current jupyter package versions is when installing jupyter components via mamba/conda instead of pip.

this behaviour isn't specific to notebook, i could also get it on lab and on jupyter console, though jupyter-console is a little different where it give this message and then the kernel becomes unresponsive. I don't get this behaviour when using ipython console directly.

.../python-lib/lib/python3.10/site-packages/jupyter_console/ptshell.py:787: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kernel_is_complete` to False.
  warn('The kernel did not respond to an is_complete_request. '

i tried increasing c.ZMQTerminalInteractiveShell.kernel_is_complete_timeout = 1 to 60s for console and that didn't prevent the kernel from becoming unresponsive.

any thoughts/suggestions on where this issue should be raised? something extra might be being done on the conda-forge jupyter feedstock that's missing from pip/pypi.

[RRosio]

Hi @miloaissatu thank you for the additional information! Have you tried to run this in ipython and have you seen this same issue there? If so, this might be better suited for the ipython repo.

[miloaissatu]

couldn't reproduce it on ipython, something to do with jupyter-client<-pyzmq->ipykernel.

looks like anything newer than jupyter-client==7.2.0+pyzmq==22.2.1 has this issue

[miloaissatu]

update - can't reproduce the issue after installing the respective OS' zeromq-devel or equivalent, then pyzmq with pip install --no-binary

had some indication from doing a diff on conda's pyzmq vs pip's downloaded pyzmq and also pyzmq's documentation which suggests building from source rather than prebuilt binaries if facing issues. https://github.com/zeromq/pyzmq#building-and-installation

[RRosio]

Hi @miloaissatu, thank you for the further updates! Does this seem to be similar to the issue reported at https://github.com/jupyter/notebook/issues/6721?

[JasonWeill]

@miloaissatu Any update regarding @RRosio 's question last week?

[miloaissatu]

Hi @RRosio , @JasonWeill , no that issue doesn't seem related imo.

I haven't been able to reproduce my problem since having pip install pyzmq via source instead of the prebuilt binaries.

[RRosio]

Hi @miloaissatu thank you for your response. Is this an issue that is then solvable by updating our dependency on pyzmq or might it be an issue already reported in pyzmq?

[miloaissatu]

Perhaps if your dependency specifies it requires pyzmq but to build from source rather than use pre-built binaries, not sure about what needs to be in the python package but in a requirements file it would be

pyzmq
--no-binary=pyzmq

its not an issue in pyzmq, not one i could recognise anyway. only hint was on the pyzmq pypi and docs page. excerpt from https://pypi.org/project/pyzmq/

If the wheel doesn't work for some reason, or you want to force pyzmq to be compiled (this is often preferable if you already have libzmq installed and configured the way you want it), you can force installation with:

[bubbawilliams]

on on

There's a typo.

Happy to Help,

Very New Guy to GitHub