Kernel crashes randomly: `IndexError: pop from an empty deque`

lsouchet commented 5 months ago

Description

Hello, We encounter a random error which occurs from time to time on our shared jupyterhub instance. After some time of running many jupyterlab notebooks, the kernel tends the crash always at the same cell of a notebook(which changes from a crash occurrence to another). Note that the same cell works in other runs, and that the error does not occur on the same cell or cell content after the jupyterhub is restarted and the error happens again. Here is the stacktrace we get:

Traceback (most recent call last):
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel_launcher.py", line 18, in <module>
    app.launch_new_instance()
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/traitlets/config/application.py", line 1075, in launch_instance
    app.start()
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 739, in start
    self.io_loop.start()
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 205, in start
    self.asyncio_loop.run_forever()
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/srv/envs/mambaforge/envs/experiments-2024-06-03/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque

To overcome the error, we need to restart the jupyterhub instance thus interrupting everybody's work.

Reproduce

Open a notebook
Run a series of cells
A popup appears telling that the kernel has crashed and need to be restarted

Expected behavior

The kernel does not crash and all cells are run successfully.

Context

Operating System and version: Debian 11 (bullseye)
Browser and version: Chrome 125, firefox 125
JupyterLab version: 4.1.2

For reference, here are all jupyter-related packages that are used to build our jupyterhub instance

jupyter-client==8.6.0 \
    --hash=sha256:0642244bb83b4764ae60d07e010e15f0e2d275ec4e918a8f7b80fbbef3ca60c7 \
    --hash=sha256:909c474dbe62582ae62b758bca86d6518c85234bdee2d908c778db6d72f39d99
    # via
    #   ipykernel
    #   jupyter-server
    #   nbclient
jupyter-core==5.7.1 \
    --hash=sha256:c65c82126453a723a2804aa52409930434598fd9d35091d63dfb919d2b765bb7 \
    --hash=sha256:de61a9d7fc71240f688b2fb5ab659fbb56979458dc66a71decd098e03c79e218
    # via
    #   ipykernel
    #   jupyter-client
    #   jupyter-server
    #   jupyterlab
    #   nbclient
    #   nbconvert
    #   nbformat
jupyter-events==0.9.0 \
    --hash=sha256:81ad2e4bc710881ec274d31c6c50669d71bbaa5dd9d01e600b56faa85700d399 \
    --hash=sha256:d853b3c10273ff9bc8bb8b30076d65e2c9685579db736873de6c2232dde148bf
    # via jupyter-server
jupyter-lsp==2.2.2 \
    --hash=sha256:256d24620542ae4bba04a50fc1f6ffe208093a07d8e697fea0a8d1b8ca1b7e5b \
    --hash=sha256:3b95229e4168355a8c91928057c1621ac3510ba98b2a925e82ebd77f078b1aa5
    # via
    #   jupyterlab
    #   jupyterlab-lsp
jupyter-resource-usage==1.0.1 \
    --hash=sha256:d722ad32fc8bfaff3f81da4f8a2202c5e5258895d546399ff0e5ddf11f56bd8e \
    --hash=sha256:ede723ebb63d531615d0da4f8769470c8ee9cc3fa18e6af9a12b70f11972cc61
    # via -r requirements.txt
jupyter-server==2.12.5 \
    --hash=sha256:0edb626c94baa22809be1323f9770cf1c00a952b17097592e40d03e6a3951689 \
    --hash=sha256:184a0f82809a8522777cfb6b760ab6f4b1bb398664c5860a27cec696cb884923
    # via
    #   jupyter-lsp
    #   jupyter-resource-usage
    #   jupyterlab
    #   jupyterlab-server
    #   notebook
    #   notebook-shim
jupyter-server-terminals==0.5.2 \
    --hash=sha256:1b80c12765da979513c42c90215481bbc39bd8ae7c0350b4f85bc3eb58d0fa80 \
    --hash=sha256:396b5ccc0881e550bf0ee7012c6ef1b53edbde69e67cab1d56e89711b46052e8
    # via jupyter-server
jupyter-telemetry==0.1.0 \
    --hash=sha256:1de3e423b23aa40ca4a4238d65c56dda544061ff5aedc3f7647220ed7e3b9589 \
    --hash=sha256:445c613ae3df70d255fe3de202f936bba8b77b4055c43207edf22468ac875314
    # via jupyterhub
jupyterhub==4.0.2 \
    --hash=sha256:2f389e7d3067e1b11bb4091719048eedecee161039fd2e5b025d031f7ab23c62 \
    --hash=sha256:d4e450eed8d90dfbcf0eca08f00f2093a0bce74dc51f7cfc0b7057f602341a1d
    # via
    #   -r requirements.txt
    #   jupyterhub-traefik-proxy
    #   sudospawner
jupyterhub-traefik-proxy==1.1.0 \
    --hash=sha256:6972d8eb788274d1d541c11cc10764d94a6829843bdb15873b60e19519bd4d5c \
    --hash=sha256:9e67eb354165b9400fb5a01c4272b4c2073dbd81fbbcd4e766ea7a62fac12cde
    # via -r requirements.txt
jupyterlab==4.1.2 \
    --hash=sha256:5d6348b3ed4085181499f621b7dfb6eb0b1f57f3586857aadfc8e3bf4c4885f9 \
    --hash=sha256:aa88193f03cf4d3555f6712f04d74112b5eb85edd7d222c588c7603a26d33c5b
    # via
    #   -r requirements.txt
    #   jupyterlab-lsp
    #   notebook
jupyterlab-lsp==5.0.3 \
    --hash=sha256:1a1c96f60202e49c28538f40b3ee19487d490a338bed88c95368073412689e22 \
    --hash=sha256:a9c8a3a646494be484d7e3174ab1d2415ed60f4773cec8479aa5ba37e5f04bc9
    # via -r requirements.txt
jupyterlab-pygments==0.3.0 \
    --hash=sha256:721aca4d9029252b11cfa9d185e5b5af4d54772bb8072f9b7036f4170054d35d \
    --hash=sha256:841a89020971da1d8693f1a99997aefc5dc424bb1b251fd6322462a1b8842780
    # via nbconvert
jupyterlab-server==2.25.3 \
    --hash=sha256:846f125a8a19656611df5b03e5912c8393cea6900859baa64fa515eb64a8dc40 \
    --hash=sha256:c48862519fded9b418c71645d85a49b2f0ec50d032ba8316738e9276046088c1
    # via
    #   jupyterlab
    #   notebook
jupyterlab-skip-traceback==5.0.0 \
    --hash=sha256:a1091b639850ef3b614b534816f38647c41a2519b4bf47ca7e7585770fb3dd9e \
    --hash=sha256:dd1acf9150c951150c6de24ebc6c3247bb7788971671158cc0915fd3310ed75b
    # via -r requirements.txt
jupyterlab-widgets==3.0.10 \

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

lsouchet commented 6 days ago

Any news on this? Is there any extra information I can try to gather in order to narrow it down?

minrk commented 3 days ago

Since this is an error in asyncio in the standard library and your Python is a few years out of date, I wonder if it may have been fixed already. If it's possible for you to try an environment with a more recent Python, that would be the first thing I would try. This same error has been reported on other libraries, such as https://github.com/Rapptz/discord.py/issues/9487 and https://github.com/erdewit/nest_asyncio/issues/11 so I'm not sure it's anything we are doing or can fix. If it is, it may relate to using nest_asyncio (if it's being used) or perhaps something involving threads.

ipython / ipykernel