Connect to an existing kernel

[hadim]

Is that possible or planned to connect to an existing kernel ?

For example, a kernel could return a JSON file with some information for the client to connect such as here http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Connecting%20with%20the%20Qt%20Console.html

Then that would be nice to have a small form in JL to be able to copy/paste the JSON file to create a new "remote kernel".

I am developing a Scijava kernel and I would like users to be able to launch the kernel inside an ImageJ/Fiji instance so it's possible to interact with the ImageJ/Fiji graphical interface directly from the notebook.

See here https://github.com/hadim/scijava-jupyter-kernel/issues/11

[blink1073]

Trying to understand the use case, do you mean that the kernel was launched from Fiji and you'd like to connect to it from a notebook in JupyterLab after that?

[hadim]

Exactly. For example my kernel would return some JSON code to copy/paste in JL (or the opposite).

So JL will create a "remote temporary kernel".

[blink1073]

The way this could be done in JupyterLab currently is to start a session using the notebook server from Fiji and then choose that session when creating a new notebook from JupyterLab that is running against the same server. The notebook server itself does not have a way to connect to a remote kernel AFAIK.

[blink1073]

@minrk, do you agree with the above?

[minrk]

@blink1073 yup. When a kernel is not started by the notebook server, some functionality is missing: interrupt, restart, etc. (this is the difference between KernelManager and KenrelClient on the Python side). However, there isn't really a technical limitation preventing the notebook server from connecting to external kernels. We would just need to handle the "interrupt and restart don't work" behavior in the UI.

This gap is precisely what KernelNanny is meant to solve, eliminating the difference in capability between kernels you started and kernels you connect to, making a 'remote' kernel equivalent to a local one.

[hadim]

Ok thanks for the explanation. But the remote kernel still need this connection_file right ? How to deal with that ?

[blink1073]

For the use case of using a shared kernel on the same server, you would post a REST API request to the notebook server to start the session from Fiji, and then that session would show up in the kernel selection dropdown list in JupyterLab automatically.

I'm not sure can reasonably support the use case of connecting to a kernel not started by the notebook server, for the reasons Min gives above. Once we have the kernel nanny, we could make a new REST endpoint that connects to a foreign kernel and then the kernel connection would be the same from the frontend (supporting interrupt and restart). That would leave getting the appropriate connection info for the remote kernel and having a user interface for connecting to a connection file or inputting the kernel JSON directly. As for how to get a connection file, I think it would be up to the kernel to provide that information if it were not started using jupyter console or the notebook server API. How are you starting the kernel from Fiji now?

[hadim]

The kernel (based on Beakerx) is added to the kernel list with conda kernelspec install. The kernel.json command launch https://github.com/hadim/scijava-jupyter-kernel/blob/master/src/main/java/org/scijava/jupyter/service/DefaultJupyterService.java#L135 in order to start the kernel.

So, for now, the kernel is started by Jupyter and outside Fiji. I want to keep this behavior in order to make it independent from Fiji.

That being said, I would like to adapt that runkernel method in order to start it without this connection_file (provided by Jupyter in the kernel.json file).

If I understand it correctly connection_file (http://jupyter-client.readthedocs.io/en/latest/kernels.html) provides an IP address + ports + signature key in order to allow communication between the kernel and Jupyter. I was wondering whether it's possible to generate the file on the kernel side and tell Jupyter about it.

The alternative (much simpler) would be to add some UI to JL (or notebook) in order to add a kernel by copy/pasting the kernel.json file (generated by my kernel).

What do you think ?

[blink1073]

I am still not understanding. Is the intent to start the kernel from within JupyterLab, get its connection info though the UI, and then input the connection info in Fiji?

[hadim]

Sorry for not being clear.

The Fiji program is juste a zip file that people extract wherever they want in their hard drives and it can also be moved or replaced​ by another one often. In consequence I don't want to write "in hard" a kernel.json file.

The sequence I see is :

• User start Fiji
• They execute a Command that will start a new kernel and display the Jupyter connection informations.
• Then they start Jupyter lab.
• Execute a widget in the menu : "Connect to a remote kernel" for example.
• The widget will popup a box where the user can paste the Jupyter connection information generated previously by Fiji.
• Then Jupyter can open a new notebook using the started Fiji remote kernel.

As an alternative, we can imagine JL displays the connection informations that is then pasted in Fiji to start a kernel.

[blink1073]

We can't support the first option until we have the kernel nanny and changes are made to the server to allow connecting to a remote kernel. The second option of displaying the connection information would still require a server change (to get the connection info), but would not require the kernel nanny. A third option in the interim is for your kernel to output its connection info as a notebook output and then copy paste that into the Fiji launcher.

[hadim]

The second option looks like a good option and should be easy to implement.

[tonyroberts]

Just to add another use case to this issue; it would be really useful to be able to specify an existing kernel in the case where Python is embedded in another application. In my specific case, I have a Python interpreter running in Excel (as an Excel add-in) and I want to be able to connect to an embedded IPython kernel running in this process.

I have been able to do this using a custom KernelProvisioner class and that works well, but if this was to be added as a feature without needing to do that, that would be even better.

In case it's useful to anyone else trying to do something similar to this the custom KernelProvisioner class I'm using is here https://github.com/pyxll/pyxll-jupyter/blob/5459a93a6cfd79d5b3f1775d19fd531c79938512/pyxll_jupyter/provisioning/existing.py. I start the embedded IPython kernel and then run jupyter lab with --KernelProvisionerFactory.default_provisioner_name=pyxll-provisioner, and have an entry point in my package linking the provisioner name to the KernelProvisioner class.

[shenh]

@tonyroberts, thanks for providing the link. I installed the package pyxll-jupyter and had error using the steps below 1 start kernel from an embedded pythonnet in .Net environment, using code below from ipykernel.kernelapp import launch_new_instance launch_new_instance(kernel_name='MyKernel', extra_arguments=[], connection_file='kernel-1084.json')

2 run the following command in cmd jupyter-lab --KernelProvisionerFactory.default_provisioner_name=pyxll-provisioner

The Jupyter-lab started and popped up a dialog to select kernel, but 'MyKernel' is not listed there. Could you kindly advise what I am missing here? Since I do not need to link pyxll, is there a way to just use existing.py and its dependencies?

Thanks.