search

spyder-ide / spyder

Official repository for Spyder - The Scientific Python Development Environment
https://www.spyder-ide.org
MIT License
8.35k stars 1.62k forks source link

Allow plugins to register frontend and kernel comms #10106

Open ChrLackner opened 5 years ago

ChrLackner commented 5 years ago

Issue Report Checklist

Problem Description

In Spyder4 the frontend communicators provide a clean way to push notifications from the kernel to the frontend. I would like to use this in a plugin, but for this some features are lacking:

A global signal, that is emitted every time a new kernel is created somewhere or attached to

This would allow the plugin to attach functions to the communicator of the kernel or to execute some code silently on kernel start. This signal should be emitted as well if a remote kernel is connected. The signal should pass the newly created kernel, so that the connected functions can modify it to their needs. Where would be the appropriate place for this signal?

Register handlers in the fronted

With the signal we can register communicators in the kernel, so we have to be able to register new handlers in the fronted as well. I guess that would be easiest if the BasePlugin implements some function to register_message_handler or so for this.

These would be my ideas for the design, do you have different ideas? I'll keep this description up to date. I would be willing to help to implement this as well.

Best Christopher

ccordoba12 commented 5 years ago

@impact27, this one is for you. I think @ChrLackner is basically asking for an API to add kernel and frontend comms, so that third-party plugin creators can benefit from the architecture you developed.

impact27 commented 5 years ago

The comms are connected by calling KernelComm.set_shell at two occasions:

So depending on what the needs are, I would say one of these three functions is a good place to emit a signal.

The frontends handlers and kernel calls are easy to do. If you have spyder_kernel_comm, you can just call register_call_handler to register a handler, and remote_call to call a function in the kernel.

Then to register something on the kernel side, you can execute get_ipython().kernel.frontend_comm.register_call_handler to register a call, and get_ipython().kernel.frontend_comm.remote_call to make a remote call.

impact27 commented 5 years ago

So if your plug in has code in spyder-kernels, you can simply add a signal in spyder/plugins/ipythonconsole/widgets/shell.py:ShellWidget.set_kernel_client_and_manager. If instead you use get_ipython().kernel.frontend_comm. to inject code in the kernel at runtime, you will also need to handle ClientWidget.restart_kernel.

ChrLackner commented 5 years ago

So basically what I want to do is to inject code at kernel startup to connect a signal of another library to call get_ipython().kernel.frontend_comm.remote_call with the signal arguments if emitted. Then I want to handle this call in the frontend.

If I understood correctly, if no shell widget is created, no frontend communicator is connected? So this wouldn't work with the notebook plugin, which not necessarily creates a ipython console connected to the kernel. Or is the notebookplugin going to use the ipythonconsole plugin, but without creating a shell widget? Wouldn't it be cleaner, if this would be in some base class, which the notebookplugin widget can derive from as well?

The question would be as well who should hold the signal and how can created shell widgets as well as my plugin access it?

impact27 commented 5 years ago

So basically what I want to do is to inject code at kernel startup to connect a signal to call get_ipython().kernel.frontend_comm.remote_call with the signal arguments if emitted. Then I want to handle this call in the frontend.

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.

The kernel client on the other hand is on the frontend side and can do that. You should use Qt signals to communicate with the kernel client. The kernel client can then send code to the kernel to be executed.

comms are specifically to communicate between the kernel and the frontend, for example the kernel client. I don't think this is what you want to do.

Maybe if you could describe why you want to do that, I could be of more help.

If I understood correctly, if no shell widget is created, no frontend communicator is connected?

Well if you don't have a kernel, there is nothing to connect to. So this wouldn't work with the notebook plugin, which not necessarily creates a ipython console connected to the kernel. Or is the notebookplugin going to use the ipythonconsole plugin, but without creating a shell widget? Wouldn't it be cleaner, if this would be in some base class, which the notebookplugin widget can derive from as well?

I am not familiar with the notebookplugin. But if you are not using the shell widget, you can still use the kernel comm to communicate with spyder, provided you use a spyder-kernels. The only reason the comms need a Shell widget is for pdb integration and runcell/ runfile functions.

The question would be as well who should hold the signal and how can created shell widgets as well as my plugin access it?

Usually, what happens is that a Shell is created for each kernel client. This kernel client is then connected to a kernel. You can be notified when a new Shell is created.

I think I would need more explanation on what and why you want to do to help you. To summarise, the way things works now are:

You don't need a Shell widget to work with kernel_clients or spyder_kernel_comm.

ChrLackner commented 5 years ago

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.

I didn't mean qt signals here. My library has it's own signals, it is loaded in the kernel and I want to attach to these signals. Sorry for being unclear here.

Maybe if you could describe why you want to do that, I could be of more help.

I've written a plugin to integrate the GUI (written in PyQt) we have developed for the finite element library NGSolve into Spyder. A short sneak peak into it is here: https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s I've done this using quite ugly monkeypatching and it doesn't work if "Enable UMR" is set in the Python Interpreter settings. So after seeing your changes I want to start a new attempt to implement it in a clean way ;) Basically I need to attach to Draw, Redraw and some other functions in NGSolve and pass their arguments to the frontend. The function, meshes,... can be pickled so they will be sent from the kernel to the frontend process. The frontend plugin handles the drawing then.

I got it working with the notebookplugin as well, but only when the notebookplugin opened a shell widget for the kernel, because all the communication is done in the shell widget. That's why I asked if that could be done in some base where the notebookplugin can derive from as well. Then I could have communicators with the notebookplugin as well to draw in my plugin.

Hope that clarifies my attempt a little, if something is still unclear let me know

Best

impact27 commented 5 years ago

The kernel is not living in the same process as the frontend. Therefore, it can not emit or receive Qt signals.

I know that. I didn't mean qt signals here. My library has it's own signals, it is loaded in the kernel and I want to attach to these signals. Sorry for being unclear here.

Maybe if you could describe why you want to do that, I could be of more help.

I've written a plugin to integrate the GUI (written in PyQt) we have developed for the finite element library NGSolve into Spyder. A short presentation is here: https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s https://www.youtube.com/watch?v=yLyz9c_ew8w&t=15s I've done this using quite ugly monkeypatching and it doesn't work if "Enable UMR" is set in the Python Interpreter settings. So after seeing your changes I want to start a new attempt to implement it in a clean way ;) Basically I need to attach to Draw, Redraw and some other functions in NGSolve and pass their arguments to the frontend. The function, meshes,... can be pickled so they will be sent from the kernel to the frontend process. The frontend plugin handles the drawing then.

I got it working with the notebookplugin as well, but only when the notebookplugin opened a shell widget for the kernel, because all the communication is done in the shell widget. That's why I asked if that could be done in some base where the notebookplugin can derive from as well. Then I could have communicators with the notebookplugin as well to draw in my plugin.

Hope that clerifies my attempt a little, if something is still unclear let me know

Best

In that case I think you just need to create a KernelComm item and connect it to the kernel_client you are using. I think the notebookplugin has a kernel_client.

ChrLackner commented 5 years ago

Ah yes. I think I didn't fully understand the KernelComm. Thanks. So the only thing I would need is to know when a new kernel_client is created somewhere and attach my KernelComm to it. At the kernel side how can I check if my communicator is available and how can I retrieve it? If this is possible, the only thing I would need is to notify my plugin somehow when a new kernel_client is created. Could we have a global signal for that?

impact27 commented 5 years ago

Ah yes. I think I didn't fully understand the KernelComm. Thanks. So the only thing I would need is to know when a new kernel_client is created somewhere and attach my KernelComm to it. At the kernel side can how can I check if my communicator is available and how can I retrieve it? If this is possible, the only thing I would need is to notify my plugin somehow when a new kernel_client is created. Could we have a global signal for that?

If you have a spyder kernel, the comm is available by calling kernel.frontend_comm. Then you can just test kernel.frontend_comm.is_open(). If you want to get notified when the comm is opening, something should be added to FrontendComm._comm_open to call a callback or something.

I think I will change the name of KernelComm.set_kernel_client(self, kernel_client) to KernelComm.open_comm(self, kernel_client).

ChrLackner commented 5 years ago

Ok I see. Then I guess the cleanest way would be if the notebookplugin creates a kernelcomm as well on kernel_client start. Then I guess the best point for emitting the signal would be at the end of KernelComm.set_kernel_client (or open_comm). Then plugins could add handlers for every kernel created.

ChrLackner commented 4 years ago

Sry I haven't had much time lately, but I was able to implement what I wanted by monkeypatching the ipyconsole.process_started and ipyconsole.connect_ecternal_kernel functions in my register_plugin function and inserting all the code I needed there. It seems to be working, thanks for the help!