search

jupyter / notebook

Jupyter Interactive Notebook
https://jupyter-notebook.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
11.77k stars 5k forks source link

Notebook 6.5.x Python Tests CI Job Failing with Jupyter_Client >= 8.0.0 #6748

Open RRosio opened 1 year ago

RRosio commented 1 year ago

Describe the bug Notebook 6.5.x Python Tests CI Job, test_connections of the KernelAPITest is failing. In recent CI environments for the Python Tests, the version of jupyter_client installed is 8.0.3 which which results in the error tornado.simple_httpclient.HTTPTimeoutError: Timeout during request when the run_sync method is called.

To Reproduce This can be locally reproduced by running the tests in notebook/services/kernels/tests/test_kernels_api.py. Observe that with a version of jupyter_client >= 8.0.0, the test_connections fails. When we install jupyter_client==7.4.9, the test suite passes.

Expected behavior Expected successful communication with the HTTP client and passing test suite.

Links Link to one instance of this failure in Python Tests CI Job https://github.com/jupyter/notebook/actions/runs/4225687193/jobs/7338276859

Desktop

Additional context Going back to the changes in jupyter_client in PR https://github.com/jupyter/jupyter_client/pull/835, the first instance of a successfully running test suite where the test failure is present is at https://github.com/jupyter/jupyter_client/pull/835/commits/00ac661591070d704ceebe1a0bf91f669e16142b

Linking https://github.com/jupyter/notebook/issues/6721 as changes in PR https://github.com/jupyter/jupyter_client/pull/835 also showed the first instance of the error described in that issue. Specifically, the commit https://github.com/jupyter/jupyter_client/pull/835/commits/ab93126452b9aea86768188270b4b1df2b01b633 is the first instance where I encountered this error message.

As @zsailer suggested in the Notebook meeting: some of the improvements for asyncio support may be clashing with the classic notebook server so pinning a version of jupyter_client and pyzmq to quickly mitigate this issue may be the next step.

smacke commented 1 year ago

On my M1 Mac, I'm getting a "production" assertion error when running a classic Jupyter notebook (not nbclassic) with newer versions of Jupyter client (see attached stacktrace.txt), which looks to be identical to the zmq message arrived on closed channel error described by other users in the Jupyter discourse: https://discourse.jupyter.org/t/jupyter-notebook-zmq-message-arrived-on-closed-channel-error/17869/17

Some users suggest that rolling back tornado to 6.1 can fix the issue, but my investigation suggests that this is only because of association with older versions of jupyter_client. I did a git bisect, and narrowed down the issue to a jupyter_client commit in the range from 6fe99ae to 99a800b, which includes the referenced commit 00ac661. (I couldn't get the exact blame because there were other issues I ran into here as well.) This is a range of ~20 or so commits introduced by PR https://github.com/jupyter/jupyter_client/pull/835.

Not sure whether this is helpful but I figured it couldn't hurt to add another data point.

panlinux commented 9 months ago

In Ubuntu and Debian, we are now also starting to see this failure in the test suite that runs at build-time. With jupyter-client 8.6.0, this test fails:

________________________ KernelAPITest.test_connections ________________________

self = <notebook.services.kernels.tests.test_kernels_api.KernelAPITest testMethod=test_connections>

    def test_connections(self):
        kid = self.kern_api.start().json()['id']
        model = self.kern_api.get(kid).json()
        self.assertEqual(model['connections'], 0)

> ws = self.kern_api.websocket(kid)
...
> return future_cell[0].result()
E tornado.simple_httpclient.HTTPTimeoutError: Timeout during request

With jupyter-client 7.4.9, it passes.

In a related project[1], @smacke did some troubleshooting and looks like they found the cause, and https://github.com/jupyter/nbconvert/pull/1972 was merged. Does that translate to what is going on here in notebook?

  1. https://github.com/jupyter/nbconvert/issues/1964
detrout commented 2 months ago

I tried fixing some issues in the Debian jupyter-notebook package and I'm definitely getting the time out error in the tests with jupyter-client 8.6.2-3, but I can't find any reference to setting the kernel_manager_class in packages that are installed.

When I downgrade to jupyter-client 7.4.9 in the test container, the test_connections test does pass.

Because the notebook completely broke because of the traitlets warn import, and the I was able to run notebooks even with this test failure I ignored the failing test to get a working package out, though it'd be good to figure what's going on.

Debian unstable current tornado version is 6.4.1-2 and pyzmq is 24.0.1.5+b2. I am also seeing the 

WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel
WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel
WARNING  NotebookApp:zmqhandlers.py:245 zmq message arrived on closed channel

during the test failure condition. While looking around for hints of what's going on I did see other bug reports referencing that issue.