async cell execution

[kmike]

Hi,

It'd be nice to be able to execute async code in IPython cells - this can allow using IPython to develop e.g. asyncio code (cell = implicit asyncio coroutine) or Scrapy spiders.

I'm having this problem in Splash Notebook kernel (https://github.com/scrapinghub/splash/blob/master/splash/kernel/kernel.py). This kernel runs a QWebKit browser in the same event loop as IPython and allows to control it using Lua scripts. Lua commands use coroutines; it means in Python land they use callbacks or deferreds. For example, there is splash:go(url) command which loads an URL in a browser and waits until all resources are loaded; this is implemented by loading URL and setting a callback for 'loadFinished' event.

Currently this is implemented using a hack: Kernel.do_execute returns a Deferred which is fired when load is finished, and Kernel.execute_request adds a callback to this Deferred to send the result back, instead of sending the result immediately.

This mostly works, but there are hacks required to make idle / busy handled correctly because ipykernel thinks results are always returned immediately. Kernel.dispatch_shell is handled incorrectly by these hacks, so sometimes kernel shows 'busy' state while it should be in 'idle' state; this breaks autocompletion for some reason (is it disabled when kernel is 'busy'?).

Sorry for a brain dump :) I don't really understand what I'm doing: what is dispatch_shell method for, what most arguments mean (parent? stream? ident? etc.). I've sent https://github.com/ipython/ipython/pull/7713 in past to make IPython internals easier to work with for such async code, but never addressed the feedback. I think async support is most important for execute messages because they have real use cases; async completion and other messages are less useful, so it is fine to handle only execute messages.

I've seen https://github.com/ipython/ipykernel/issues/21 and https://github.com/takluyver/aiokernel, but didn't understood it at all; aiokernel look very different from what I came up with, so I have no idea what is it doing, or if it is solving the same issue or not.

Thoughts? Am I on a right track? Is this feature welcome? Is there a clean(er) way to implement it?

[takluyver]

aiokernel was an experiment that I didn't take any further. You might be interested in this extension:

https://gist.github.com/takluyver/b9663b08ac9a4472afa6

That allows you to write await code in cells (using a cell magic), and it will work as if you were inside a coroutine.

[kmike]

@takluyver nice example! Before going with hacks I tried to use crochet to get a function similar to run_until_complete for Twisted, but it didn't work (something about thread handling).

[flutefreak7]

Is this fixed now?

https://blog.jupyter.org/ipython-7-0-async-repl-a35ce050f7f7