Support server-side execution

[davidbrochart]

This currently only works with jupyverse, not yet with jupyter-server. You can try it with:

pip install "jupyterlab>=4.2.0b0"
pip install "jupyverse[jupyterlab, auth]>=0.4.2"
jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Supersedes https://github.com/jupyterlab/jupyterlab/pull/15448. Closes https://github.com/jupyterlab/jupyterlab/issues/2833. Closes https://github.com/jupyterlab/jupyterlab/issues/12867.

https://github.com/jupyterlab/jupyterlab/assets/4711805/c65b5253-da77-4203-9223-b9097e6688c6

[github-actions[bot]]

:point_left: Launch a Binder on branch _davidbrochart/jupytercollaboration/server-side-execution

[davidbrochart]

I don't understand the error in Test JavaScript:

@jupyter/react-components/lib/index.js:3
export * from './wrap';
^^^^^^
SyntaxError: Unexpected token 'export'

@krassowski or @jtpio do you have an idea?

[davidbrochart]

Not sure what is happening with Test Minimum Versions, any idea @blink1073 ?

[krassowski]

We need similar change as in https://github.com/jupyterlab/jupyterlab/pull/16105

[davidbrochart]

Thanks @krassowski !

[davidbrochart]

I'm wondering if we could make a pre-release with this, since it already uses JupyterLab 4.2.0-beta.0?

[jtpio]

@davidbrochart maybe having a small screencast in the top comment could help have a better idea of what the feature looks like? Or maybe the one in https://github.com/jupyterlab/jupyterlab/pull/15448#issuecomment-1830079178 is still relevant?

[davidbrochart]

Yes, I added that screencast to the top comment :+1:

[almereyda]

This looks marvellous, congratulations. One little question about the second part, when the notebook had been reopened: Is there any other visual indication in the UI around the notebook, that there is a cell or notebook running on the server, other than the intermittent changes of the tab cross when new changes are pushed and saved right away?

My immediate intuition was to look for the play button, if it turned into some kind of pause button, but it wouldn't have changed.

Or maybe the Kernel is not actually Idle, as displayed in the status bar at the bottom, but in some kind of new state JupyterLab doesn't know about, yet? During the first run it shows as Busy and the empty circle turns into a filled circle in the top right.

[ujjawal-khare]

@davidbrochart by any chance we can release this as a pre-release. It will be really helpful for our use case.

[davidbrochart]

This looks marvellous, congratulations. One little question about the second part, when the notebook had been reopened: Is there any other visual indication in the UI around the notebook, that there is a cell or notebook running on the server, other than the intermittent changes of the tab cross when new changes are pushed and saved right away?

The execution state is currently not handled by this new execution mode but we are working on it, see https://github.com/jupyter-server/jupyter_ydoc/pull/227.

[davidbrochart]

@davidbrochart by any chance we can release this as a pre-release. It will be really helpful for our use case.

Yes that's the plan, give us a little bit more time :smile: Also, beware that this currently only works with Jupyverse.

[davidbrochart]

I'm having a last-minute thought. With the discussions around execution state, I'm wondering if having a REST API for cell execution is the best. Instead, we could for instance take advantage of the "execution_state" property of a cell to trigger execution. A client wishing to execute a cell would set this field to "busy", if currently in "idle". This would make server-side execution transport-agnostic (not specific to a REST API). What do you think?

[krassowski]

In jupyter-server we can (in principle) forbid execution for specific users as we have an authorization layer (https://jupyter-server.readthedocs.io/en/latest/operators/security.html#authorization). It is easy to wrap a REST API with authorization. Can this be also done easily with shared model modifications without increasing its complexity?

Another concern, possibly easier to address is how to execute a cell multiple times (well of course that could be a list of requests, and each request could be numbered by the client).

[davidbrochart]

In jupyter-server we can (in principle) forbid execution for specific users as we have an authorization layer (https://jupyter-server.readthedocs.io/en/latest/operators/security.html#authorization). It is easy to wrap a REST API with authorization. Can this be also done easily with shared model modifications without increasing its complexity?

The Y WebSocket is also authorized, so we could use permissions to accept/discard incoming shared model changes. In pycrdt there is experimental support for model validation (e.g. using Pydantic), which goes in the same direction. But I admit this is unknown territory.

Another concern, possibly easier to address is how to execute a cell multiple times (well of course that could be a list of requests, and each request could be numbered by the client).

I was thinking that if a client sets the status to busy (meaning it requests an execution) while the kernel is already executing the cell, that would have no effect. So there would be no such thing as queuing cell executions.

[davidbrochart]

But since this is an experimental feature, only working in jupyverse, and under a flag, maybe it's okay to merge. We can still iterate in the future.

[davidbrochart]

Now that we depend on NPM packages that are in pre-release, does it imply that we should also make a pre-release, say v2.0.12b0, or can it just be v2.0.12?

[jtpio]

Since it adds a new feature, maybe it can be bumped to 2.1.0b0?

[davidbrochart]

OK to bump, be should it be a beta?

[jtpio]

No strong opinions. If it doesn't change the default behavior of the jupyter-collaboration package it could even be a 2.1.0 release directly. Otherwise an alpha would be fine, followed by a final release if everything is ok.

[krassowski]

I would say a beta, but can we also get https://github.com/jupyterlab/jupyter-collaboration/pull/275 and my upcoming PR which splits backend and frontend in?

[krassowski]

I think I can have the repo revamp ready for review today, but version drift will just add more work for me.

[davidbrochart]

OK, let's wait then, there is no hurry.

[rishabh-dream11]

@davidbrochart Just checking to see if there is consensus on the solution and can be used, I have a product requirement pending on it, hence asking for the same. We use jupyter-server and have not tried jupyverse.

[imerelli]

Hi, is there any update regarding this feature, which I personally consider very important? When will it be integrated into the official release of Jupyter?

[krassowski]

Good question. I think with https://github.com/jupyterlab/jupyter-collaboration/pull/307 and https://github.com/datalayer/jupyter-server-nbmodel it will become at least possible to use it via an extension

[vinaypoduval19]

@krassowski How can I test this feature out? The above extension seems to not be available on pypi

[krassowski]

@vinaypoduval19 I am impatient too, but the jupyter-server-nbmodel extension is in a third-party organization and contributors here do not have a control on when it gets released. I am grateful to @datalayer for developing the extension and I'm trying to be helpful by leaving a review on https://github.com/datalayer/jupyter-server-nbmodel/pull/20 and opening https://github.com/datalayer/jupyter-server-nbmodel/pull/21 but this is as much as I can do.

[b10902118]

This currently only works with jupyverse, not yet with jupyter-server. You can try it with:

pip install "jupyterlab>=4.2.0b0"
pip install "jupyverse[jupyterlab, auth]>=0.4.2"
jupyverse --set kernels.require_yjs=true --set jupyterlab.server_side_execution=true

Supersedes jupyterlab/jupyterlab#15448. Closes jupyterlab/jupyterlab#2833. Closes jupyterlab/jupyterlab#12867.

Peek.2023-11-28.16-25.mp4

This seems not working now. I tried the exact version == or >=, also with jupyter-collaboration installed but still not working.