Event loop integration

[SimonDanisch]

When I'm using a Julia library that starts a loop with @async, the python REPL will never yield to it. It seems like we need to insert a Main.eval("yield()") into the python repl loop. Could be an easy PR, but I'm not really sure where that would go!

[tkf]

I highly recommend using IPython instead of bare Python REPL: https://pyjulia.readthedocs.io/en/latest/limitations.html#ctrl-c-does-not-work-terminates-the-whole-python-process

If you are using IPython and PyJulia, I think the easiest way to integrate it is to actually start from Julia REPL and hop in IPython via https://github.com/tkf/IPython.jl because the event loop is already integrated: https://github.com/tkf/IPython.jl/blob/f34fd122f19528416776a72b8491a6b15fe5061e/src/ipython_jl/ipyext.py#L45-L60

If you must start from Python, maybe you can use https://github.com/tkf/ipyjulia_hacks

[stevengj]

Couldn't pyjulia just use the Python threading module, something like:

import threading
import time
def yieldloop():
    while True:
        julia.yield()
        time.sleep(10e-6)
threading.Thread(daemon=True, target=yieldloop).start()

?

[tkf]

Wouldn't it break because libjulia is not thread-safe? Or is it OK after Julia 1.3?

[SimonDanisch]

I highly recommend using IPython instead of bare Python REPL:

Well, this is for wrapping my Julia library in Python for others, so I won't really know from where they'll be using my package ;)

[tkf]

If you understand the limitation of Julia runtime, you can recommend your users to use IPython or IPython.jl. You can at least be prepared to not be surprised when your users report that Ctrl+C killed their Python REPL.

The only actionable item I can think of is to copy asyncio integration https://github.com/tkf/ipyjulia_hacks/blob/f5d0e0f91da5a2bca7a987de2e634e04f88c76c4/src/ipyjulia_hacks/ipy/magic.py#L33-L46 from ipyjulia_hacks so that @async works at least inside ipykernel out-of-the-box. The tricky part is that we still support Python 2... Maybe we should just unsupport Python < 3.5.

[stevengj]

Wouldn't it break because libjulia is not thread-safe? Or is it OK after Julia 1.3?

I thought that Python only runs one thread at a time because of the GIL, so they are effectively green threads and hence thread-safety is not an issue?

[tkf]

Python thread is real OS thread so for Julia it still means that there is a callback from a different thread. It at least didn't work in the past: https://github.com/JuliaPy/pyjulia/issues/132#issuecomment-388605547

Also there could be a third-party external library called from Python in the main thread that releases the GIL and calls some libjulia functions. So, I don't think it's entirely safe to poll in background thread even if Julia runtime allows callback from another thread. (Although we can just say "don't do this" in this scenario.)

[SimonDanisch]

Btw, after quite a bit of fiddling, this was the only solution that worked for me:

import asyncio, itertools
async def julia_yielder():
    for i in itertools.count():
        Main.eval("yield()")
        await asyncio.sleep(1/30)

asyncio.ensure_future(julia_yielder())

No idea why the internet is full of 1 mio solutions, that all don't seem to work^^

[tkf]

Note that asyncio.ensure_future solution only works when asyncio event loop is running. So, it works inside Jupyter but not in normal Python REPL or terminal IPython.

[SimonDanisch]

Uhm... is it really not possible to run a loop async in Python without starting the event loop, which will block?

[tkf]

It's not possible. Event loop in Python is optional. Since various different libraries implement their own event loop, you have to use a specific entry point to launch a background task for each library.