Enhancement: Expose a global IKernelConnection to output renderers

[rchiodo]

Right now we generate a global WidgetManager and a proxy kernel for ipywidgets to work.

For customers that don't want to write ipywidgets but still need kernel communication (one such one is Synapse) they really only need the IKernelConnection.

We should expose this as a global in our preloads for when a kernel runs.

Additionally, we should check what Jupyter classic did to expose this before. It would be nice to follow the same API pattern (exactly if possible) with the intent to make it easier to port jupyter extensions to VS code/AZNB

[rchiodo]

An add on to this would be to expose this same IKernelConnection on the extension host side with an api like:

getKernel(uri: Document): Promise<IKernelConnection | undefined>

[duburcqa]

Is there any plan to implement this at some point ?

[rchiodo]

It's not in our near future plans (no milestone set) but it's needed by a partner team so it's likely it will get implemented at some point.

[DonJayamanne]

@duburcqa What's your user case for this feature?

[duburcqa]

@DonJayamanne I'm developing the robotic simulator jiminy using a webgl viewer based on treejs called meshcat. In ipython i can instantiate a iframe, that is opening a connection with the kernel in javascript so that the viewer and the main python thread can communicate back and forth. Currently, I can create the iframe in VSCode and used python to communication with the viewer, but not the other way around. I really need it to handle basic events such as waiting for the meshes to load before capturing the frame programmatically.

[DonJayamanne]

@duburcqa We've created some API that exposes the Kernels and their API to external extensions. This is still a work in progress and we don't have any samples for for communicating with the kernel from webviews and using widgets (it will require quite a bit of work). Please feel free to have a look at the API exposed by the extension.

[duburcqa]

@DonJayamanne Interesting ! Is there any documentation somewhere ? I'm relying on one javascript method in particular:

window.parent.Jupyter.notebook.kernel.comm_manager.new_comm

[DonJayamanne]

window.parent.Jupyter.notebook.kernel.comm_manager.new_comm

We do not expose this anywhere. Currently there are no plans to expose this, as this could pose a security risk (allowing any extension to access the kernel and execute code - an issue when dealing with remote jupyter kernels)

[duburcqa]

Ok i see. I was expecting this. No problem. Thank you anyway !

[DonJayamanne]

I'll see what can be done, I think we can use the same model we have today (exposing 3rd party API) to expose these kernels in output renderers.

Today when extensions attempt to access the kernel API user is prompted to authorize this. We can do the same when extensions attempt to access the kernel API in renderers (via something like a token).

[DonJayamanne]

@duburcqa Do you have a VS Code extension that ships the code running in the renderers?

[duburcqa]

@DonJayamanne No, it is a python package to install via pip:

pip install jiminy_py

Then, a basic jupyter noteboox example can be found here. Running it will embedded the renderer in a cell using a HTML iframe, and the latter while try to open a connection with the kernel through javascript in order to send simple information such as acknowledgements. Here is the embedded HTML code I'm referring to. Note that this approach is not working for jupyterlab for a similar security concern. I'm open to developing a proper extension module but I have zero knowledge on this and I don't know where to start (I'm a Python/C++ guy).

Thank you for paying attention to my problem!

Update: I can ensure you there is a real need for this. There is a real community of people trying to improve the notebook integration of a 3D renderer for robotics application, in particular meshcat. I believe making it work properly in VSCode may be helping to get closer to this goal. For reference: https://github.com/RobotLocomotion/drake/issues/12645

[duburcqa]

After digging a little bit in other issues, I'm facing the issue than here: https://github.com/microsoft/vscode-jupyter/issues/1477

[DonJayamanne]

I'm facing the issue than here: #1477

Not following you, is iminy_py also opening up sockets and communicating with the kernel via its own sockets? It not, then its not the same as #1477

[duburcqa]

It is partially the same. I'm not communicating with the kernel via my own sockets, but they are relying on IPython.notebook.kernel.comm_manager just like me. Besides, I'm also using a non-standard technique to communicate with the kernel, although I'm not opening dedicated socket but rather hijacking the ones opened by jupyter to avoid having to handle it myself).

[DonJayamanne]

From what I can tell, Jupyter lab renderers cannot access the kernel unless the renderers are Jupyter Lab specific renderers. Such as the following https://github.com/jupyterlab/jupyterlab/tree/master/packages/vdom

I.e. the widget gets the Kernel as a ctor argument.

[duburcqa]

hum. I tried to look a bit but my knowledge in javascript and jupyter is too limited to build something similar.

[DonJayamanne]

@duburcqa If you own the widget code (which happens to be the case), then you should be able to access the kernel directly as follows: this.widget_manager or this.model.widget_manager

I.e. there's no need to expose the kernel directly. This issue was created to address the issue where widgets are not necessarily owned/nor created by extension authors.

Summary:

• Widgets have access to the kernels

[duburcqa]

I want it to be compatible with cloud notebook providers such as colab, kaggle or deepnote. So developping a widget is not an option.

[DonJayamanne]

So developping a widget is not an option

We try to support compatibility with Jupyter lab and classic Jupyter notebook. From what you're saying you aren't willing to create a Jupyter notebook/lab widget. I'd recommend you consider that as it's already an industry standard, and as mentioned earlier, exposing kennels another way could lose security issues, basically any J's code from other extensions could execute code on local and remote machines.

Then again theres the whole security issue.

[duburcqa]

I see your point. I will think about it. Yet, for you to know, recently I managed to get it to work in VSCose and Jupyter by connecting to the kernel websocket manually. So the current implementation of VSCose notebooks is not preventing this at all, just making it more complicated.

[DonJayamanne]

Closing this issue as we expose an API for users to access the kernel connection. If users which to access the kernel, they could, send messages back to the extension host from the webview .

[Conchylicultor]

@duburcqa I'm having the same issue. Do you have an example of how to connect to the kernel without creating a custom widget ?

My use-case is to run IPython.notebook.kernel.execute for bidirectional Python communication.