jupyter / notebook

Jupyter Interactive Notebook
https://jupyter-notebook.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
11.51k stars 4.81k forks source link

Jupyter Notebook ‘zmq message arrived on closed channel’ Error #6721

Open caferistretto opened 1 year ago

caferistretto commented 1 year ago

Describe the bug Hello!

I’ve recently begun to experience near constant zmq/tornado errors when running Jupyter notebook from my conda environment (Jupyter, conda env, and traceback details below). When the error occurs, code cells in the Jupyter Notebook fail to either run entirely and/or the cells run but the output does not get printed to the Notebook.

The same error also occurs while using JupyterLab but seemingly less frequently. On occasion, Jupyter will run for several hours without exhibiting the issue while in other instances the error starts occurring immediately after a Notebook is started.

I would really appreciate any suggestions for resolving the issue. Thanks so much!

To Reproduce

  1. Activate conda env
  2. Start jupyter notebook by calling jupyter-notebook in bash
  3. Open an .ipynb notebook and (preferably) run multiple cells at once using either 'Run All' or 'Run All Above'
  4. If the error traceback (copied below) does not appear immediately, it will arise after the Notebook is used for a few minutes.

Expected behavior Connection to the notebook is maintained and output of stdout is printed/redirected to the notebook

Desktop (please complete the following information):

macOS v12.6.3 Firefox v109.0.1

Additional context

Output of jupyer --version

IPython : 8.9.0
ipykernel : 6.21.1
ipywidgets : 8.0.4 jupyter_client : 8.0.2 jupyter_core : 5.2.0 jupyter_server : 2.2.1 jupyterlab : 3.5.3 nbclient : 0.7.2 nbconvert : 7.2.9 nbformat : 5.7.3 notebook : 6.5.2 qtconsole : 5.4.0 traitlets : 5.9.0

Output of conda --list

anyio 3.6.2 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py311h5547dcb_3 conda-forge arrow-cpp 11.0.0 h694c41f_2_cpu conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge attrs 22.2.0 pyh71513ae_0 conda-forge aws-c-auth 0.6.23 hb2f45f0_1 conda-forge aws-c-cal 0.5.20 h3aaaed9_4 conda-forge aws-c-common 0.8.9 hb7f2c08_0 conda-forge aws-c-compression 0.2.16 h980c674_1 conda-forge aws-c-event-stream 0.2.18 h947c3fe_2 conda-forge aws-c-http 0.7.3 h9091ec1_1 conda-forge aws-c-io 0.13.14 h666a9c8_3 conda-forge aws-c-mqtt 0.8.6 h2814666_1 conda-forge aws-c-s3 0.2.3 h24f324c_3 conda-forge aws-c-sdkutils 0.1.7 h980c674_1 conda-forge aws-checksums 0.1.14 h980c674_1 conda-forge aws-crt-cpp 0.18.16 ha2465d3_13 conda-forge aws-sdk-cpp 1.10.57 hcf18aa8_2 conda-forge babel 2.11.0 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.2 pyha770c72_0 conda-forge bleach 6.0.0 pyhd8ed1ab_0 conda-forge blosc 1.21.2 hebb52c4_0 conda-forge brotli 1.0.9 hb7f2c08_8 conda-forge brotli-bin 1.0.9 hb7f2c08_8 conda-forge brotlipy 0.7.0 py311h5547dcb_1005 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge c-ares 1.18.1 h0d85af4_0 conda-forge ca-certificates 2022.12.7 h033912b_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py311ha86e640_3 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.1.2 pyhd8ed1ab_0 conda-forge contourpy 1.0.7 py311hd2070f0_0 conda-forge cryptography 39.0.0 py311h61927ef_0 conda-forge curl 7.87.0 h6df9250_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge debugpy 1.6.6 py311h814d153_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 hf0c8a7f_0 conda-forge flit-core 3.8.0 pyhd8ed1ab_0 conda-forge fonttools 4.38.0 py311h5547dcb_1 conda-forge freetype 2.12.1 h3f81eb7_1 conda-forge future 0.18.3 pyhd8ed1ab_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge gflags 2.2.2 hb1e8313_1004 conda-forge git 2.39.1 pl5321h0195497_0 conda-forge glib 2.74.1 hbc0c0cd_1 conda-forge glib-tools 2.74.1 hbc0c0cd_1 conda-forge glog 0.6.0 h8ac2a54_0 conda-forge gst-plugins-base 1.21.3 h37e1711_1 conda-forge gstreamer 1.21.3 h1d18e73_1 conda-forge h5py 3.8.0 nompi_py311h777de7a_100 conda-forge hdf5 1.12.2 nompi_h48135f9_101 conda-forge icu 70.1 h96cf925_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.0.0 pyha770c72_0 conda-forge importlib_metadata 6.0.0 hd8ed1ab_0 conda-forge importlib_resources 5.10.2 pyhd8ed1ab_0 conda-forge ipykernel 6.21.1 pyh736e0ef_0 conda-forge ipython 8.9.0 pyhd1c38e8_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 8.0.4 pyhd8ed1ab_0 conda-forge jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e hac89ed1_2 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jupyter 1.0.0 py311h6eed73b_8 conda-forge jupyter_client 8.0.2 pyhd8ed1ab_0 conda-forge jupyter_console 6.4.4 pyhd8ed1ab_0 conda-forge jupyter_core 5.2.0 py311h6eed73b_0 conda-forge jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge jupyter_server 2.2.1 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab 3.5.3 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.19.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 3.0.5 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.4 py311hd2070f0_1 conda-forge krb5 1.20.1 h049b76e_0 conda-forge lcms2 2.14 h29502cd_1 conda-forge lerc 4.0.0 hb486fe8_0 conda-forge libabseil 20220623.0 cxx17_h844d122_6 conda-forge libaec 1.0.6 hf0c8a7f_1 conda-forge libarrow 11.0.0 hf6017ac_2_cpu conda-forge libblas 3.9.0 16_osx64_openblas conda-forge libbrotlicommon 1.0.9 hb7f2c08_8 conda-forge libbrotlidec 1.0.9 hb7f2c08_8 conda-forge libbrotlienc 1.0.9 hb7f2c08_8 conda-forge libcblas 3.9.0 16_osx64_openblas conda-forge libclang 13.0.1 default_he082bbe_0 conda-forge libcrc32c 1.1.2 he49afe7_0 conda-forge libcurl 7.87.0 h6df9250_0 conda-forge libcxx 14.0.6 hccf4f1f_0 conda-forge libdeflate 1.17 hac1461d_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libevent 2.1.10 h7d65743_4 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 11_3_0_h97931a8_27 conda-forge libgfortran5 11.3.0 h082f757_27 conda-forge libglib 2.74.1 h4c723e1_1 conda-forge libgoogle-cloud 2.7.0 hb5e37a9_0 conda-forge libgrpc 1.51.1 h1ddfa78_1 conda-forge libiconv 1.17 hac89ed1_0 conda-forge liblapack 3.9.0 16_osx64_openblas conda-forge libllvm13 13.0.1 h64f94b2_2 conda-forge libnghttp2 1.51.0 he2ab024_0 conda-forge libogg 1.3.4 h35c211d_1 conda-forge libopenblas 0.3.21 openmp_h429af6e_3 conda-forge libopus 1.3.1 hc929b4f_1 conda-forge libpng 1.6.39 ha978bb4_0 conda-forge libpq 15.1 h3640bf0_3 conda-forge libprotobuf 3.21.12 hbc0c0cd_0 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libsqlite 3.40.0 ha978bb4_0 conda-forge libssh2 1.10.0 h47af595_3 conda-forge libthrift 0.16.0 h16802d8_2 conda-forge libtiff 4.5.0 hee9004a_2 conda-forge libutf8proc 2.8.0 hb7f2c08_0 conda-forge libvorbis 1.3.7 h046ec9c_0 conda-forge libwebp-base 1.2.4 h775f41a_0 conda-forge libxcb 1.13 h0d85af4_1004 conda-forge libzlib 1.2.13 hfd90126_4 conda-forge llvm-openmp 15.0.7 h61d9ccf_0 conda-forge lz4-c 1.9.4 hf0c8a7f_0 conda-forge markupsafe 2.1.2 py311h5547dcb_0 conda-forge matplotlib 3.6.3 py311h6eed73b_0 conda-forge matplotlib-base 3.6.3 py311h2bf763f_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mistune 2.0.4 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.31 hc4b2c72_0 conda-forge mysql-libs 8.0.31 h8658499_0 conda-forge nbclassic 0.5.1 pyhd8ed1ab_0 conda-forge nbclient 0.7.2 pyhd8ed1ab_0 conda-forge nbconvert 7.2.9 pyhd8ed1ab_0 conda-forge nbconvert-core 7.2.9 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 7.2.9 pyhd8ed1ab_0 conda-forge nbformat 5.7.3 pyhd8ed1ab_0 conda-forge ncurses 6.3 h96cf925_1 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge notebook 6.5.2 pyha770c72_1 conda-forge notebook-shim 0.2.2 pyhd8ed1ab_0 conda-forge nspr 4.35 hea0b92c_0 conda-forge nss 3.78 ha8197d3_0 conda-forge numexpr 2.8.3 py311hd84f3f5_1 conda-forge numpy 1.24.2 py311ha9d2c9f_0 conda-forge openjpeg 2.5.0 h13ac156_2 conda-forge openssl 3.0.7 hfd90126_2 conda-forge orc 1.8.2 ha9d861c_2 conda-forge packaging 23.0 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py311hd84f3f5_0 conda-forge pandoc 2.19.2 h694c41f_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 h1c4e4bc_0 conda-forge perl 5.32.1 2_h0d85af4_perl5 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.4.0 py311h5bae705_1 conda-forge pip 23.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge pooch 1.6.0 pyhd8ed1ab_0 conda-forge prometheus_client 0.16.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.36 pyha770c72_0 conda-forge prompt_toolkit 3.0.36 hd8ed1ab_0 conda-forge psutil 5.9.4 py311h5547dcb_0 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyarrow 11.0.0 py311hb17b21d_2_cpu conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.14.0 pyhd8ed1ab_0 conda-forge pymatreader 0.0.30 pyhd8ed1ab_0 conda-forge pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyqt 5.15.7 py311h8771221_3 conda-forge pyqt5-sip 12.11.0 py311h78fa98d_3 conda-forge pyrsistent 0.19.3 py311h5547dcb_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytables 3.7.0 py311ha9b306f_3 conda-forge python 3.11.0 he7542f4_1_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.4 pyhd8ed1ab_0 conda-forge python_abi 3.11 3_cp311 conda-forge pytz 2022.7.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py311h5547dcb_5 conda-forge pyzmq 25.0.0 py311habfacb3_0 conda-forge qt-main 5.15.6 haeff654_6 conda-forge qtconsole 5.4.0 pyhd8ed1ab_0 conda-forge qtconsole-base 5.4.0 pyha770c72_0 conda-forge qtpy 2.3.0 pyhd8ed1ab_0 conda-forge re2 2023.02.01 hf0c8a7f_0 conda-forge readline 8.1.2 h3899abd_0 conda-forge requests 2.28.2 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge scikit-learn 1.2.1 py311h087fafe_0 conda-forge scipy 1.10.0 py311h939689b_2 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 67.1.0 pyhd8ed1ab_0 conda-forge sip 6.7.7 py311h814d153_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.9 h225ccf5_2 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge sqlite 3.40.0 h9ae0607_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.13.5 py311hd5badaa_2 conda-forge terminado 0.17.1 pyhd1c38e8_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h5dbffcc_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tornado 6.2 py311h5547dcb_1 conda-forge tqdm 4.64.1 pyhd8ed1ab_0 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge tzdata 2022g h191b570_0 conda-forge urllib3 1.26.14 pyhd8ed1ab_0 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.5.1 pyhd8ed1ab_0 conda-forge wheel 0.38.4 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.5 pyhd8ed1ab_0 conda-forge xmltodict 0.13.0 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 h35c211d_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge zeromq 4.3.4 he49afe7_1 conda-forge zipp 3.12.1 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hfd90126_4 conda-forge zstd 1.5.2 hbc0c0cd_6 conda-forge

Error printed to Jupyter’s console:

[W TIMESTAMP NotebookApp] zmq message arrived on closed channel

Python traceback:

[E 09:29:38.352 NotebookApp] Uncaught exception in ZMQStream callback Traceback (most recent call last): File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback f = callback(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback return callback(self, msg) ^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply super()._on_zmq_reply(stream, msg) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply self.write_message(msg, binary=isinstance(msg, bytes)) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 339, in write_message return self.ws_connection.write_message(message, binary=binary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1086, in write_message fut = self._write_frame(True, opcode, message, flags=flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1061, in _write_frame return self.stream.write(frame) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 546, in write self._handle_write() File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 976, in _handle_write self._write_buffer.advance(num_bytes) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 182, in advance assert 0 < size <= self._size AssertionError [E 09:29:38.357 NotebookApp] Uncaught exception in zmqstream callback Traceback (most recent call last): File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events self._handle_recv() File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv self._run_callback(callback, msg) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback f = callback(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback return callback(self, msg) ^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply super()._on_zmq_reply(stream, msg) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply self.write_message(msg, binary=isinstance(msg, bytes)) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 339, in write_message return self.ws_connection.write_message(message, binary=binary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1086, in write_message fut = self._write_frame(True, opcode, message, flags=flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1061, in _write_frame return self.stream.write(frame) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 546, in write self._handle_write() File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 976, in _handle_write self._write_buffer.advance(num_bytes) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 182, in advance assert 0 < size <= self._size AssertionError [E 09:29:38.358 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.. at 0x1086751c0>) Traceback (most recent call last): File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/ioloop.py", line 740, in _run_callback ret = callback() ^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 705, in self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events self._handle_recv() File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv self._run_callback(callback, msg) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback f = callback(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback return callback(self, msg) ^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply super()._on_zmq_reply(stream, msg) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply self.write_message(msg, binary=isinstance(msg, bytes)) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 339, in write_message return self.ws_connection.write_message(message, binary=binary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1086, in write_message fut = self._write_frame(True, opcode, message, flags=flags) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/websocket.py", line 1061, in _write_frame return self.stream.write(frame) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 546, in write self._handle_write() File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 976, in _handle_write self._write_buffer.advance(num_bytes) File "/Users/data/miniconda3/envs/da/lib/python3.11/site-packages/tornado/iostream.py", line 182, in advance assert 0 < size <= self._size AssertionError

RRosio commented 1 year ago

Hi @caferistretto thank you for reporting this issue. Do you have any security software running? In #2627, it was reported that security software was blocking network traffic.

caferistretto commented 1 year ago

Hi @RRosio - No, I don't have any security or firewall software running on my mac. Thanks so much for the help!

RRosio commented 1 year ago

No problem @caferistretto, thank you for your response! Do you by any chance have some adblock enabled on your browser? I am only able to reproduce this issue in Firefox when an adblock extension is enabled.

caferistretto commented 1 year ago

@RRosio I do have have uBlock Origin installed! Curiously, it doesn't seem to be reporting that any elements on the page are being block, but a test with it disabled has kept the Jupyter Notebook zmq issue from occurring. I will update this issue after several days of testing if this was indeed an issue with adblock. :)

nagydavid commented 1 year ago

Hi All,

I just would like to tie in the bug report at vscode, since some of us experienced through vscode. #

DonJayamanne commented 1 year ago

@nagydavid thanks for commenting here, just wanted to verify that @nagydavid can replicate this issue even outside vscode. @RRosio here are the logs from @nagydavid https://github.com/microsoft/vscode-jupyter/issues/12775#issuecomment-1425506777

nagydavid commented 1 year ago

@DonJayamanne Thank you to bring the logs here too. @RRosio unfortunately I could reproduce the error with the adblocker disabled. Also VPN turned off. Only Microsoft Defender is enabled.

Architecture: Mac OS, M1 arm64, but it happens in Rosetta2 mode as well.

relevant libraries: jupyter==1.0.0 jupyter-console==6.4.4 jupyter-events==0.6.3 jupyter_client==8.0.2 jupyter_core==5.2.0 jupyter_server==2.2.1 jupyter_server_terminals==0.4.4 jupyterlab-pygments==0.2.2 jupyterlab-widgets==1.1.1 ipykernel==6.21.1 ipython==8.9.0 ipython-genutils==0.2.0 ipywidgets==7.7.2 notebook==6.5.2 notebook_shim==0.2.2

Recent log:

> ~/Documents/work/OWF.AML/.venv/bin/python ~/.vscode/extensions/ms-toolsai.jupyter-2023.2.1000421009/pythonFiles/printJupyterDataDir.py
debug 10:53:17.218: getDataDirsImpl, Class name = Lu, completed in 57ms, has a truthy return value, Arg 1: {"resource":{"$mid":1,"fsPath":"/Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb","external":"file:///Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb","path":"/Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb","scheme":"file"},"interpreter":{"id":"/Users/davidnagy/Documents/work/OWF.AML/.venv/bin/python","sysPrefix":"/Users/davidnagy/Documents/work/OWF.AML/.venv","envPath":{"$mid":1,"fsPath":"/Users/davidnagy/Documents/work/OWF.AML/.venv","path":"/Users/davidnagy/Documents/work/OWF.AML/.venv","scheme":"file"},"displayPath":{"$mid":1,"fsPath":"/Users/davidnagy/Documents/work/OWF.AML/.venv","path":"/Users/davidnagy/Documents/work/OWF.AML/.venv","scheme":"file"},"envName":".venv","uri":{"$mid":1,"fsPath":"/Users/davidnagy/Documents/work/OWF.AML/.venv/bin/python","external":"file:///Users/davidnagy/Documents/work/OWF.AML/.venv/bin/python","path":"/Users/davidnagy/Documents/work/OWF.AML/.venv/bin/python","scheme":"file"},"displayName":".venv","envType":"PipEnv","version":{"major":3,"minor":10,"patch":9,"raw":"3.10.9 (main, Feb  7 2023, 11:28:57) [Clang 14.0.0 (clang-1400.0.29.202)]"}}}
debug 11:05:35.723: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:05:35.724: Execute Cell 0 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:05:35.725: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:05:35.726: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:05:35.727: Start cell 0 execution @ 1676023535727
debug 11:05:56.747: End cell 0 execution @ 1676023556747, started @ 1676023535727, elapsed time = 21.02s
debug 11:05:56.747: Cell 0 executed with state Success
debug 11:05:56.747: VSCodeNotebookController::handleExecution, Class name = kv, completed in 21024ms, has a falsy return value
debug 11:05:59.316: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:05:59.317: Execute Cell 1 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:05:59.317: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:05:59.317: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:05:59.318: Start cell 1 execution @ 1676023559318
debug 11:05:59.407: End cell 1 execution @ 1676023559407, started @ 1676023559318, elapsed time = 0.089s
debug 11:05:59.407: Cell 1 executed with state Success
debug 11:05:59.407: VSCodeNotebookController::handleExecution, Class name = kv, completed in 91ms, has a falsy return value
debug 11:06:01.509: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:06:01.510: Execute Cell 2 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:01.510: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:01.510: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:06:01.512: Start cell 2 execution @ 1676023561511
debug 11:06:01.580: End cell 2 execution @ 1676023561580, started @ 1676023561511, elapsed time = 0.069s
debug 11:06:01.580: Cell 2 executed with state Success
debug 11:06:01.580: VSCodeNotebookController::handleExecution, Class name = kv, completed in 71ms, has a falsy return value
debug 11:06:04.454: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:06:04.455: Execute Cell 3 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:04.455: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:04.455: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:06:04.456: Start cell 3 execution @ 1676023564456
debug 11:06:07.967: End cell 3 execution @ 1676023567967, started @ 1676023564456, elapsed time = 3.511s
debug 11:06:07.967: Cell 3 executed with state Success
debug 11:06:07.967: VSCodeNotebookController::handleExecution, Class name = kv, completed in 3513ms, has a falsy return value
debug 11:06:09.839: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:06:09.839: Execute Cell 4 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:09.839: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:09.840: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:06:09.840: Start cell 4 execution @ 1676023569840
debug 11:06:09.908: End cell 4 execution @ 1676023569908, started @ 1676023569840, elapsed time = 0.068s
debug 11:06:09.908: Cell 4 executed with state Success
debug 11:06:09.908: VSCodeNotebookController::handleExecution, Class name = kv, completed in 69ms, has a falsy return value
debug 11:06:13.130: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:06:13.130: Execute Cell 5 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:13.130: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:13.131: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:06:13.131: Start cell 5 execution @ 1676023573131
debug 11:06:31.622: End cell 5 execution @ 1676023591622, started @ 1676023573131, elapsed time = 18.491s
debug 11:06:31.622: Cell 5 executed with state Success
debug 11:06:31.623: VSCodeNotebookController::handleExecution, Class name = kv, completed in 18493ms, has a falsy return value
debug 11:06:40.676: VSCodeNotebookController::handleExecution, Class name = kv (started execution)
debug 11:06:40.676: Execute Cell 6 /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:40.676: start the kernel, options.disableUI=false for /Users/davidnagy/Documents/work/OWF.AML/darts_test.ipynb
debug 11:06:40.677: Start Jupyter Session in kernel.ts with disableUI = false
debug 11:06:40.678: Start cell 6 execution @ 1676023600678
[E 11:06:41.693 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[I 11:06:41.699 NotebookApp] Starting buffering for 56c35693-cdf1-4fbe-84f5-60688466a40e:0d2be91a-1fa5-4303-832b-d3fbbc3887a6
[E 11:06:41.740 NotebookApp] Uncaught exception in zmqstream callback
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events
        self._handle_recv()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[E 11:06:41.741 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x10718a440>)
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/ioloop.py", line 740, in _run_callback
        ret = callback()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 705, in <lambda>
        self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events
        self._handle_recv()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[I 11:06:41.743 NotebookApp] Restoring connection for 56c35693-cdf1-4fbe-84f5-60688466a40e:0d2be91a-1fa5-4303-832b-d3fbbc3887a6
tonyfast commented 1 year ago

seems like a bunch of folks are reporting this message on the discourse board too https://discourse.jupyter.org/t/jupyter-notebook-zmq-message-arrived-on-closed-channel-error/17869/5

caferistretto commented 1 year ago

Hi all - as a quick update: disabling adblock has not stopped the zmq/tornado event loop error from occurring in Jupyter Notebook. As in the past, the error may appear either right away once the first few cells of the notebook are run or after several minutes of use.

I have tried downgrading Python to versions 3.9 and 3.10 without resolution. Haven't tried downgrading tornado/zmq yet as doing so in isolation with conda becomes difficult given all the dependencies yoked to those packages.

nagydavid commented 1 year ago

@RRosio Can we ask an update on the bug?

DonJayamanne commented 1 year ago

Ok, I’m also now running into this issue on my Mac M1 in Jupyter Notebook (without vscode)

thejokenott commented 1 year ago

I'm also experiencing the issue (Mac ARM64, vscode).

JoepVanlier commented 1 year ago

Experiencing this or a similar issue on Windows 10 w/ Chrome 109.0.5414.120 with a fresh environment installing only notebook and matplotlib.

For me reverting to jupyter_client=7.4.9 seems to have resolved the errors as a temporary stopgap.

[E 13:58:25.233 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "C:\Conda\envs\test_notebooks\lib\site-packages\zmq\eventloop\zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\zmq\eventloop\zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\notebook\services\kernels\handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\notebook\base\zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\iostream.py", line 546, in write
        self._handle_write()
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "C:\Conda\envs\test_notebooks\lib\site-packages\tornado\iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
jupyter --version
Selected Jupyter core packages...
IPython          : 8.10.0
ipykernel        : 6.21.2
ipywidgets       : not installed
jupyter_client   : 8.0.2
jupyter_core     : 5.2.0
jupyter_server   : 2.2.1
jupyterlab       : not installed
nbclient         : 0.7.2
nbconvert        : 7.2.9
nbformat         : 5.7.3
notebook         : 6.5.2
qtconsole        : not installed
traitlets        : 5.9.0
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h8d17308_3    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.2             pyha770c72_0    conda-forge
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                hcfcfb64_8    conda-forge
brotli-bin                1.0.9                hcfcfb64_8    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2022.12.7            h5b45459_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h628cb3f_3    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.2              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.7           py310h232114e_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
debugpy                   1.6.6           py310h00ffb61_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
flit-core                 3.8.0              pyhd8ed1ab_0    conda-forge
fonttools                 4.38.0          py310h8d17308_1    conda-forge
freetype                  2.12.1               h546665d_1    conda-forge
gettext                   0.21.1               h5728263_0    conda-forge
glib                      2.74.1               h12be248_1    conda-forge
glib-tools                2.74.1               h12be248_1    conda-forge
gst-plugins-base          1.22.0               h001b923_0    conda-forge
gstreamer                 1.22.0               h6b5321d_0    conda-forge
icu                       70.1                 h0e60522_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
importlib_metadata        6.0.0                hd8ed1ab_0    conda-forge
importlib_resources       5.10.2             pyhd8ed1ab_0    conda-forge
intel-openmp              2023.0.0         h57928b3_25922    conda-forge
ipykernel                 6.21.2             pyh025b116_0    conda-forge
ipython                   8.10.0             pyh08f2357_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.18.2             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   hcfcfb64_3    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter_client            8.0.2              pyhd8ed1ab_0    conda-forge
jupyter_core              5.2.0           py310h5588dad_0    conda-forge
jupyter_events            0.6.3              pyhd8ed1ab_0    conda-forge
jupyter_server            2.2.1              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.4           py310h232114e_1    conda-forge
krb5                      1.20.1               heb0366b_0    conda-forge
lcms2                     2.14                 ha5c8aab_1    conda-forge
lerc                      4.0.0                h63175ca_0    conda-forge
libblas                   3.9.0              16_win64_mkl    conda-forge
libbrotlicommon           1.0.9                hcfcfb64_8    conda-forge
libbrotlidec              1.0.9                hcfcfb64_8    conda-forge
libbrotlienc              1.0.9                hcfcfb64_8    conda-forge
libcblas                  3.9.0              16_win64_mkl    conda-forge
libclang                  15.0.7          default_h77d9078_1    conda-forge
libclang13                15.0.7          default_h77d9078_1    conda-forge
libdeflate                1.17                 hcfcfb64_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libglib                   2.74.1               he8f3873_1    conda-forge
libhwloc                  2.8.0                h039e092_1    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              16_win64_mkl    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libpng                    1.6.39               h19919ed_0    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libsqlite                 3.40.0               hcfcfb64_0    conda-forge
libtiff                   4.5.0                hf8721a0_2    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp-base              1.2.4                h8ffe710_0    conda-forge
libxcb                    1.13              hcd874cb_1004    conda-forge
libxml2                   2.10.3               hc3477c8_0    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markupsafe                2.1.2           py310h8d17308_0    conda-forge
matplotlib                3.7.0           py310h5588dad_0    conda-forge
matplotlib-base           3.7.0           py310h51140c5_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   2.0.5              pyhd8ed1ab_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbclassic                 0.5.1              pyhd8ed1ab_0    conda-forge
nbclient                  0.7.2              pyhd8ed1ab_0    conda-forge
nbconvert                 7.2.9              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.2.9              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.2.9              pyhd8ed1ab_0    conda-forge
nbformat                  5.7.3              pyhd8ed1ab_0    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
notebook                  6.5.2              pyha770c72_1    conda-forge
notebook-shim             0.2.2              pyhd8ed1ab_0    conda-forge
numpy                     1.24.2          py310hd02465a_0    conda-forge
openjpeg                  2.5.0                ha2aaf27_2    conda-forge
openssl                   3.0.8                hcfcfb64_0    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pandoc                    2.19.2               h57928b3_1    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                h17e33f8_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.4.0           py310hdbb7713_1    conda-forge
pip                       23.0               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.0.0              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
prometheus_client         0.16.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.36             pyha770c72_0    conda-forge
psutil                    5.9.4           py310h8d17308_0    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.14.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7          py310h1fd54f2_3    conda-forge
pyqt5-sip                 12.11.0         py310h00ffb61_3    conda-forge
pyrsistent                0.19.3          py310h8d17308_0    conda-forge
python                    3.10.9          h4de0772_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.2             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.6              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pywin32                   304             py310h00ffb61_2    conda-forge
pywinpty                  2.0.10          py310h00ffb61_0    conda-forge
pyyaml                    6.0             py310h8d17308_5    conda-forge
pyzmq                     25.0.0          py310hcd737a0_0    conda-forge
qt-main                   5.15.8               h720456b_6    conda-forge
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                67.3.1             pyhd8ed1ab_0    conda-forge
sip                       6.7.7           py310h00ffb61_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tbb                       2021.7.0             h91493d7_1    conda-forge
terminado                 0.15.0          py310h5588dad_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tornado                   6.2             py310h8d17308_1    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodedata2              15.0.0          py310h8d17308_0    conda-forge
vc                        14.3                hb6edc58_10    conda-forge
vs2015_runtime            14.34.31931         h4c5c07a_10    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.5.1              pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
winpty                    0.4.3                         4    conda-forge
xorg-libxau               1.0.9                hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zipp                      3.13.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.2                h12be248_6    conda-forge
RRosio commented 1 year ago

Thanks everyone for the updates about experiencing this issue! I am not able to consistently replicate this issue in my environment, so it is also difficult to pinpoint an apparent package of interest. I am wondering if there is a specific set of package versions during which you began experiencing this issue, is this issue always present or does it happen intermittently?

As @JoepVanlier mentioned trying another version of jupyter_client and as mentioned in this post on discourse of someone trying different version of tornado, do these seem to make a difference for any others?

I've also checked a few other repos of interest (jupyter_server, tornado and pyzmq) for similar recent issues but have found none.

subhashbylaiah commented 1 year ago

I am facing this problem as well, on Mac M2. Trying to pin jupyter_client=7.4.9 did not help.

johneriksloper commented 1 year ago

I am facing this problem as well, on Mac M2. Trying to pin jupyter_client=7.4.9 did not help.

I am on M1. Reverting back to jupyter-client==7.3.2 and tornado=6.1 worked for me. Obviously not a long term solution.

subhashbylaiah commented 1 year ago

I am facing this problem as well, on Mac M2. Trying to pin jupyter_client=7.4.9 did not help.

I am on M1. Reverting back to jupyter-client==7.3.2 and tornado=6.1 worked for me. Obviously not a long term solution.

Thanks for the suggestion. Sadly, this still didnt help.

reyvababtista commented 1 year ago

Facing the same problem here with Mac M1 Pro. I wrote the details in the vscode-jupyter repo here: microsoft/vscode-jupyter#12775

nagydavid commented 1 year ago

with jupyter_client==7.4.9, the error remains.

ug 10:49:49.247: VSCodeNotebookController::handleExecution, Class name = Av, completed in 1103ms, has a falsy return value
[E 10:49:49.223 NotebookApp] Uncaught exception in ZMQStream callback
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[E 10:49:49.223 NotebookApp] Uncaught exception in zmqstream callback
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events
        self._handle_recv()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[E 10:49:49.223 NotebookApp] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x113077370>)
    Traceback (most recent call last):
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/ioloop.py", line 740, in _run_callback
        ret = callback()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 705, in <lambda>
        self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events
        self._handle_recv()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv
        self._run_callback(callback, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
        f = callback(*args, **kwargs)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
        return callback(self, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
        super()._on_zmq_reply(stream, msg)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
        self.write_message(msg, binary=isinstance(msg, bytes))
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
        return self.ws_connection.write_message(message, binary=binary)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
        fut = self._write_frame(True, opcode, message, flags=flags)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
        return self.stream.write(frame)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
        self._handle_write()
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
        self._write_buffer.advance(num_bytes)
      File "/Users/davidnagy/Documents/work/OWF.AML/.venv/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
        assert 0 < size <= self._size
    AssertionError
[I 10:49:49.235 NotebookApp] Starting buffering for 9d2f4b61-6824-4cbf-b450-272ef09c497c:8277db58-7932-449d-b8fe-36997e82083b
[I 10:49:49.251 NotebookApp] Restoring connection for 9d2f4b61-6824-4cbf-b450-272ef09c497c:8277db58-7932-449d-b8fe-36997e82083b

with jupyter-client==7.3.2 and tornado=6.1 After the code execution, the kernel stuck in restart phase.

tomaskrehlik commented 1 year ago

This bug is hitting me too, notebook replicating: Reproduce.ipynb.zip I suspect some interaction of Matplotlib, Mac and Jupyter. Only happens early in the session for me with big enough plot. E.g. if you play around and comment out some of the plt.text in the notebook, the bug does not trigger. Replicates in both Jupyter Notebook and VS Code.

smason commented 1 year ago

Have just hit on this as well and noticed you're having trouble replicating. Have cut things down to:

python3.10 -m venv venv
source venv/bin/activate
pip install -U notebook matplotlib ipywidgets
jupyter notebook &> outerr.log

Running under Python 3.10.9 under Linux 6.1.12 with /usr/lib/libzmq.so.5.2.4 (i.e. up to date ArchLinux), and here's outerr.log and pip freeze output for reference. I then ran the following Python code:

%matplotlib notebook
import matplotlib.pyplot as plt
plt.scatter([0,1],[0,1])

I also get a similar traceback after installing ipympl and using %matplotlib widget magic. At some points while figuring out what was needed to replicate I had to use the "move tool" and drag the plot a few pixels before it crashed.

Hope this helps!

kalpanachinnappan commented 1 year ago

Our team is seeing same issues. Running %matplotlib widget and interacting with the plot for a few seconds causes the "move" and crop actions to freeze, making the plot not interactive anymore. During the plot crash we see these errors in server logs:

[E 2023-02-21 16:57:57.904 SingleUserNotebookApp ioloop:760] Exception in callback functools.partial(<function ZMQStream._update_handler.<locals>.<lambda> at 0x7fd01d6570a0>)
Traceback (most recent call last):
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/ioloop.py", line 740, in _run_callback
ret = callback()
File "/opt/conda/envs/default/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 705, in <lambda>
self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
File "/opt/conda/envs/default/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 621, in _handle_events
self._handle_recv()
File "/opt/conda/envs/default/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 650, in _handle_recv
self._run_callback(callback, msg)
File "/opt/conda/envs/default/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 584, in _run_callback
f = callback(*args, **kwargs)
File "/opt/conda/envs/default/lib/python3.10/site-packages/zmq/eventloop/zmqstream.py", line 308, in stream_callback
return callback(self, msg)
File "/opt/conda/envs/default/lib/python3.10/site-packages/notebook/services/kernels/handlers.py", line 572, in _on_zmq_reply
super()._on_zmq_reply(stream, msg)
File "/opt/conda/envs/default/lib/python3.10/site-packages/notebook/base/zmqhandlers.py", line 256, in _on_zmq_reply
self.write_message(msg, binary=isinstance(msg, bytes))
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/websocket.py", line 339, in write_message
return self.ws_connection.write_message(message, binary=binary)
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/websocket.py", line 1086, in write_message
fut = self._write_frame(True, opcode, message, flags=flags)
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/websocket.py", line 1061, in _write_frame
return self.stream.write(frame)
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/iostream.py", line 546, in write
self._handle_write()
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/iostream.py", line 976, in _handle_write
self._write_buffer.advance(num_bytes)
File "/opt/conda/envs/default/lib/python3.10/site-packages/tornado/iostream.py", line 182, in advance
assert 0 < size <= self._size
AssertionError
tomaskrehlik commented 1 year ago

Looking into #6748 , doing pip install --upgrade "pyzmq<25" "jupyter_client<8" solves my issue right now.

caferistretto commented 1 year ago

Hi all - I wanted to share a temporary solution that on my end resolved the ZMQ/tornado errors (at least temporarily until a final fix is implemented)

Step 1: Create new conda env with Python set to v3.10.x. We must pin python to v3.10 else one cannot install Tornado v6.1

conda create -n temp_jupyter python=3.10

Step 2: Edit the contents of the pinned packages file

touch ~/miniconda3/envs/temp_jupyter/conda-meta/pinned
nano ~/miniconda3/envs/temp_jupyter/conda-meta/pinned

In pinned put the following:

python 3.10.x
tornado ==6.1

Step 3: Install jupyter notebook and update conda env

conda activate temp_jupyter
conda install jupyter
conda update --all

Step 4: Reinstall whatever packages you need into the new env using conda install.

I hope this helps others!

AckslD commented 1 year ago

I ran into the same issue and can confirm that downgrading jupyter-client to 7.4.9 seem to have solved it

mkxxq commented 1 year ago

Looking into #6748 , doing pip install --upgrade "pyzmq<25" "jupyter_client<8" solves my issue right now.

solved my problem by following this method. Thank you very much

smacke commented 1 year ago

On my M1 Mac, I'm getting a "production" assertion error when running a classic Jupyter notebook (not nbclassic) with newer versions of Jupyter client (see attached stacktrace.txt), which looks to be identical to the zmq message arrived on closed channel error described by other users here and in the Jupyter discourse: https://discourse.jupyter.org/t/jupyter-notebook-zmq-message-arrived-on-closed-channel-error/17869/17

Some users suggested there that rolling back tornado to 6.1 can fix the issue, but my investigation suggests that this is only because of association with older versions of jupyter_client. I did a git bisect, and narrowed down the issue to a jupyter_client commit in the range from 6fe99ae to 99a800b. (I couldn't get the exact blame because there were other issues I ran into here as well.) This is a range of ~20 or so commits introduced by PR https://github.com/jupyter/jupyter_client/pull/835.

Not sure whether this is helpful but I figured it couldn't hurt to add another data point.

smacke commented 1 year ago

Did a bit more digging; it seems like this bug was introduced when the utility function run_sync was rewritten to no longer use nest_asyncio. This function was moved from jupyter_client to jupyter_core recently, so fixing this bug will require a PR to https://github.com/jupyter/jupyter_core.

sk1p commented 1 year ago

Did a bit more digging; it seems like this bug was introduced when the utility function run_sync was rewritten to no longer use nest_asyncio. This function was moved from jupyter_client to jupyter_core recently, so fixing this bug will require a PR to https://github.com/jupyter/jupyter_core.

So it possibly boils down to this issue?

https://github.com/tornadoweb/tornado/issues/2871#issuecomment-638577266

[...] In general, you cannot interact with any tornado objects which use the event loop, from a different thread than the one they were created on [...]

run_sync is using a _TaskRunner which "runs an asyncio event loop on a background thread". So if that interacts with some tornado objects, that could be the cause for this issue.

smacke commented 1 year ago

So it possibly boils down to this issue?

Not sure -- I only know that, if I replace the new run_sync with the old one, the issue goes away for me. No idea whether it's related to tornado threading issues; it seems odd that the reports are exclusively coming from M1 and M2 mac users.

EDIT: that is not true according to below data points from folks on Windows / Linux.

I lack context on the reasoning behind the change, but I'm guessing it has something to do with removing the need for nest_asyncio, which has some compat issues depending on the version. I actually ran into some issues with nest_asyncio in my work on ipyflow but I was too scared to try and implement my own nested coroutine logic and ended up just monkey patching it.

Monkey patching libraries that themselves do monkey patching is of course not ideal but it's the best I could come up with.

To work around the current issue of zmq message arrived on closed channel, I ended up writing a server extension for ipyflow that monkey patches _TaskRunner.run in jupyter_core.utils to use the original nest_asyncio approach. Anybody that installs ipyflow will get the workaround regardless of whether they use the ipyflow kernel or ipykernel, and shouldn't need to pin jupyter_[client|core|server] or tornado or anything, but same caveat appplies: not a long term fix:

pip install --upgrade ipyflow jupyter_client jupyter_server jupyter_core tornado

cc also @blink1073 and @davidbrochart for visibility; I think you both have a lot more context on issues around nest_asyncio and nested coroutines in general :)

sk1p commented 1 year ago

it seems odd that the reports are exclusively coming from M1 and M2 mac users.

Just as another data point, I've seen the issue personally on both M1 Mac and also Windows/Linux systems.

AckslD commented 1 year ago

I'm not, I'm on a thinkpad with arch linux and seeing this issue with jupyter-client newer than 7.4.9

smacke commented 1 year ago

@sk1p @AckslD does the issue go away with the workaround I proposed?

pip install --upgrade ipyflow jupyter_client jupyter_server jupyter_core tornado
jamescooke commented 1 year ago

@smacke Could you confirm the versions of those packages that you'd like us to try? I'm on linux and the previous pin of jupyter-client<=7.4.9 worked for me - am happy to try new fixes 👍🏻

smacke commented 1 year ago

@jamescooke I'm hoping that you shouldn't need to pin anymore and that the latest versions would work, thanks to the server extension that comes with ipyflow.

So theoretically, my hope is that everything should work after running

pip install --upgrade ipyflow jupyter_client jupyter_server jupyter_core tornado
AckslD commented 1 year ago

@smacke I can't reproduce the error anymore but also as before the kernel takes a lot of time to start (30 seconds). If I either downgrade jupyter-client to 7.4.9 or install the packages with your command above this is fixed. However, normally I don't use ipyflow.

smacke commented 1 year ago

@AckslD you don't have to use the ipyflow kernel to pick up the fix since it's installed as a server extension; I just added it to the ipyflow package since this bug is easier to reproduce for users of the ipyflow kernel. You should be able to use ipykernel as usual with the latest versions of these packages, provided the latest ipyflow is also installed. (At least that's my hypothesis)

That said, if you do happen to try out ipyflow, please don't hesitate to open issues on https://github.com/ipyflow/ipyflow if there are improvements you'd like to see :)

BenYork726 commented 1 year ago

@smacke worked for me! Thank you so much mate, back to blessed normal errors <3

jamescooke commented 1 year ago

@smacke Looks like it's working for me on linux 🎉

Here's what I've done (I'm using pip-tools under the hood):

Nice one - thanks 👍🏻 . I've even got shift+tab function signature hints back again, which is great 🙌🏻

smacke commented 1 year ago

Glad to hear! Opened https://github.com/jupyter/jupyter_core/pull/353 on jupyter_core so hopefully core Jupyter maintainers will have eyes on it at some point.

jamescooke commented 1 year ago

Unfortunately, I might have spoken prematurely because I'm getting some RecursionError instances happening after the notebook has been running for a while. Am not sure exactly what triggers it:

handle: <Handle _patched_patch_task.<locals>.step()>                                                                                                                                                               
Traceback (most recent call last):                                                                       
  File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run                                                                                                                                             
    self._context.run(self._callback, *self._args)                                                       
  File "/home/parallels/active/mxcld_notebooks/venv310/lib/python3.10/site-packages/ipyflow/kernel/patched_nest_asyncio.py", line 21, in step 
    step_orig(task, exc)                                                                                 
  File "/home/parallels/active/mxcld_notebooks/venv310/lib/python3.10/site-packages/ipyflow/kernel/patched_nest_asyncio.py", line 21, in step
    step_orig(task, exc)                                                                                 
  File "/home/parallels/active/mxcld_notebooks/venv310/lib/python3.10/site-packages/ipyflow/kernel/patched_nest_asyncio.py", line 21, in step
    step_orig(task, exc)                                                                                 
  [Previous line repeated 1486 more times]                                                               
  File "/home/parallels/active/mxcld_notebooks/venv310/lib/python3.10/site-packages/nest_asyncio.py", line 196, in step
    step_orig(task, exc)                                                                                                                                                                                           
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 216, in __step                                                                                                                                           
    if self.done():                                                                                                                                                                                                
  File "/usr/local/lib/python3.10/asyncio/futures.py", line 185, in done                                 
    return self._state != _PENDING                                                                                                                                                                                 
RecursionError: maximum recursion depth exceeded in comparison  
smacke commented 1 year ago

Hmm, yeah nest_asyncio is probably not the best thing to rely on (to say nothing of a patched version of it). Well it was worth a try -- you'll probably want to go back to the approach of pinning jupyter_client (and probably disable the ipyflow server extension while you're at it):

jupyter server extension disable ipyflow

EDIT: the fix I pushed to ipyflow seems to fix this, so hopefully it should be gone after pip install --upgrade ipyflow

smacke commented 1 year ago

Hmm I do see one potential bug that could account for this -- if nest_asyncio.apply() is called multiple times, we could end up with this kind of error. I just pushed a fix to ipyflow but I'm not sure whether it's the issue, if you're brave you can try it out:

pip install --upgrade ipyflow

EDIT: I was able to reproduce the original issue, and the fix I pushed to ipyflow seems to fix this, so hopefully it should be gone after pip install --upgrade ipyflow

jamescooke commented 1 year ago

@smacke Have been brave and run the new ipyflow == 0.0.173 for a whole day of work with no recursion errors or other clearly noticeable crashes 🙌🏻

moonstarsky37 commented 1 year ago

I recently encountered this issue while setting up a container. After spending some time researching package dependencies, I found a workaround that currently allows me to run it successfully without using any nb extensions. Here are the additional commands I execute after installing jupyter:

pip install jupyter-server==2.3.0 jupyter-client==7.3.2

There might be better approaches available, but this is one of the few methods that has worked for me so far.

yacchin1205 commented 1 year ago

We had the same issue with our custom image and had a hard time, but we finally found the cause and resolved in our case and share it here for your reference.

In our case, it seems to have been caused by using the blocking(non-async) version of Kernel Manager. We used our own KernelManager subclass of jupyter_client.ioloop.IOLoopKernelManager to customize the shutdown process of a kernel. It still seemed to work fine at this time, but we found that by not specifying this as the kernel manager, this issue stoppped being observed. So I implemented a (non-blocking) Kernel Manager using jupyter_client.ioloop import AsyncIOLoopKernelManager and started using it, and we no longer encounter the problem.

Currently, the docker-stacks (especially jupyter/scipy-notebook) image does not seem to have this problem, and in our case, the problem seemed to continuously occur as extensions that use blocking version (old/non-asynchronous) code are enabled.

chebee7i commented 1 year ago

Any progress on this? Hitting this issue with:

$ jupyter --version
Selected Jupyter core packages...
IPython          : 8.12.2
ipykernel        : 6.23.3
ipywidgets       : 8.0.6
jupyter_client   : 8.3.0
jupyter_core     : 5.3.0
jupyter_server   : 2.6.0
jupyterlab       : 3.6.3
nbclient         : 0.8.0
nbconvert        : 7.6.0
nbformat         : 5.9.0
notebook         : 6.5.4
qtconsole        : 5.4.3
traitlets        : 5.9.0

$ conda list | grep tornado
tornado                   6.3.2            py38h01eb140_0    conda-forge
alexhartl commented 1 year ago

This appears to be a concurrency issue that occurs if _handle_write is called multiple times in parallel, so that the same data is consumed twice. Protecting the 4 relevant lines by a mutex as done in this diff works around the issue for me. Unfortunately I don't know enough about how io and parallelism in jupyter and tornado is (supposed to be) working to develop a fix.

AaronM04 commented 1 year ago

I saw an assertion error in tornado while running my notebook, and I can confirm that upgrading ipyflow fixed the issue.

Note: I was directed here from https://github.com/tornadoweb/tornado/issues/3281#issuecomment-1599918832

jamescooke commented 1 year ago

Just a stupid idea, probably not worth my typing, but has anyone tried running with python -O to strip the assertion error?