PyGObject and asyncio

[lazka]

On the PyGObject IRC channel there is regular talk on how to best integrate with asyncio and if PyGObject should provide better integration out of the box.

What are your thoughts on integrating gblub into PyGObject?

(related bug report: https://bugzilla.gnome.org/show_bug.cgi?id=791591)

[gbtami]

It would be awesome! Now I have to use my "poor man integration": https://github.com/pychess/pychess/blob/master/pychess#L224 in PyChess because gbulb has unresolved issues.

If gbulb can be integrated into PyGObject I hope it will get more developers/contributors and may boost it's development.

[stuaxo]

As someone that has used both libraries: +1 one on this.

I'd imagine there may be potential to make the integration better / easier. There aren't a huge amount of developers on both projects, joining forces probably makes sense.

[nhoad]

I'm certainly interested in this! It'd be good to get some more people in the code base.

At the moment the code that's in master hasn't been released - it's essentially a reimplementation that gives a more full-fledged event loop that adds Windows support (instead of hacking support in on top of the Unix event loop), so I'm understandably paranoid about it. The end result is the same on Linux, however the biggest issue with it at the moment is that subprocesses don't work on Windows because non-blocking streams aren't supported by GObject's IOChannels (on Windows only, I think?). I don't know if this is something that better integration could help with, but more people working on the problem would help. I've contemplated just doing the release without subprocess support on Windows until I can figure out a workaround.

Also @gbtami, which unresolved issues are you referencing? The subprocess one?

Also also, if people want development to be a bit faster, they could help me out - I'm just one person working on this project for fun when I have some spare time, getting more people in who actually use it would be great.

[gbtami]

@nathan-hoad yes #24 Telling the truth my "poor man integration" which uses asyncio loop next to glib loop works OK. But it"s just a workaround of course. Regarding contributing to gbulb you are absolutely right and I have to apologize. Last year I started to dig into windows subprocess issue and tried to figure out how quamash solved it, but I felt it needs more windows knowledge than I have. See https://github.com/harvimt/quamash/blob/master/quamash/_windows.py

[lazka]

That's great to hear!

I'll try to add an asyncio page to the PyGObject website which points to gbulb and shows some examples. Maybe that will get more people interested.

One long term issue, if we ever try to move some of the code into PyGObject itself is that it would have to be licensed under the LGPLv2.1+ (or something compatible at least, like MIT) - Any thoughts on that?

[gzxu]

By the way, it will be great to add these two helper functions, though they are really simple. PyGObject lacks some sweet syntax sugars :laughing:

def connect_async(self, detailed_signal, handler_async, *args):
    def handler(self, *args):
        asyncio.ensure_future(handler_async(self, *args))
    self.connect(detailed_signal, handler, *args)

GObject.GObject.connect_async = connect_async

def wrap_asyncio(target, method, *, priority=False):
    async_begin = getattr(target, method + '_async')
    async_finish = getattr(target, method + '_finish')

    def wrapper(self, *args):
        def callback(self, result, future):
            future.set_result(async_finish(self, result))
        future = asyncio.get_event_loop().create_future()
        if priority:
            async_begin(self, *args, GLib.PRIORITY_DEFAULT, None, callback, future)
        else:
            async_begin(self, *args, None, callback, future)
        return future
    setattr(target, method + '_asyncio', wrapper)

An example usage will be

#!/usr/bin/env python3
import gi  # NOQA: E402
gi.require_versions({
    'Gtk': '3.0',
    'Soup': '2.4'
})  # NOQA: E402

import sys
import asyncio

import gbulb
from gi.repository import Gtk, Gio, Soup, GLib, GObject

def connect_async(self, detailed_signal, handler_async, *args):
    def handler(self, *args):
        asyncio.ensure_future(handler_async(self, *args))
    self.connect(detailed_signal, handler, *args)

GObject.GObject.connect_async = connect_async

def wrap_asyncio(target, method, *, priority=False):
    async_begin = getattr(target, method + '_async')
    async_finish = getattr(target, method + '_finish')

    def wrapper(self, *args):
        def callback(self, result, future):
            future.set_result(async_finish(self, result))
        future = asyncio.get_event_loop().create_future()
        if priority:
            async_begin(self, *args, GLib.PRIORITY_DEFAULT, None, callback, future)
        else:
            async_begin(self, *args, None, callback, future)
        return future
    setattr(target, method + '_asyncio', wrapper)

wrap_asyncio(Soup.Request, 'send')
wrap_asyncio(Gio.InputStream, 'read_bytes', priority=True)
wrap_asyncio(Gio.InputStream, 'close', priority=True)

class Window(Gtk.ApplicationWindow):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, title='Async Window', **kwargs)
        self.connect_async('realize', self.on_realize)

    async def on_realize(self, *args, **kwargs):
        button = Gtk.Button(label="Get")
        button.connect_async("clicked", self.on_button_clicked)

        entry = Gtk.Entry()
        entry.set_text('https://httpbin.org/get')

        text_view = Gtk.TextView()

        grid = Gtk.Grid()
        grid.attach(button, 1, 0, 1, 1)
        grid.attach(entry, 0, 0, 1, 1)
        grid.attach(text_view, 0, 1, 2, 1)
        self.add(grid)

        self.show_all()
        self.entry = entry
        self.text_view = text_view

    async def on_button_clicked(self, widget):
        session = Soup.Session()
        uri = Soup.URI.new(self.entry.get_text())
        request = session.request_http_uri('GET', uri)
        stream = await request.send_asyncio()
        data = await stream.read_bytes_asyncio(4096)
        self.text_view.get_buffer().set_text(data.get_data().decode())
        await stream.close_asyncio()

class Application(Gtk.Application):
    window = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, application_id=None, **kwargs)

    def do_activate(self):
        if not self.window:
            self.window = Window(application=self)
        self.window.present()

if __name__ == '__main__':
    gbulb.install(gtk=True)
    asyncio.get_event_loop().run_forever(Application(), sys.argv)

It is really fascinating to run every line of code inside just one thread, including networking and GUI :tada: :tada:

[nhoad]

Hello, I wanted to update everyone who's been involved in gbulb and I figured this was the best place to do it. Due to personal reasons and goings on in my life, I'm not able to give gbulb the attention it deserves. If people are interested in taking ownership and maintaining it, please discuss on here so we can come to an agreement, and I'll transfer ownership.

[gzxu]

:+1: I think that gbulb is an excellent initiative, but IMHO overriding the undocumented SelectorEventLoop is not a good idea. I am submitting a patch wxWidgets/Phoenix#1103 to add asyncio support to WxPython, but currently I don't have time to write documentation and tests. BTW I am using my own implementation to use coroutines in my Python GTK applications.

[nhoad]

The SelectorEventLoop is by far the most common event loop. Nearly all of the documentation relates to it.

[gbtami]

@gzxu https://docs.python.org/3/library/asyncio-eventloop.html#event-loop-implementations

[gzxu]

Oh well, I meant that symbols starts with _ are extensively accessed here, which are private and may be subject to change. :man_shrugging:

[benzea]

Soo, I randomly wondered about this again and played a bit with it over the weekend.

I think one of the major things we want here, is to turn allow GLib async functions to be run using asyncio using a nice syntax, my idea for that is the following:

• If we detect a callback that can be converted into a future (i.e. only one and has a closure and no destroy notify and we can guess the _finish function), then set the default value to gi.FutureCallback(finish_func) for it and gi.FutureCancellable() for the cancellable.
• Should the user pass additional user arguments, return them as part of the future result. My motivation for that is mostly to have a well defined behaviour
• HOWEVER: This would technically be an API break, as the defaults are usually None right now. In all cases where the return value is void it does not matter though, as the future is invisible and will just finish by itself. i.e. I think it is acceptable to do this.

Also, I guess we need to get GBulb into a shape where it is mergeable into pygobject.

For fun, I hacked up glib-asyncio to dispatch from the GLib mainloop. It is kind of neat as it is simple, but I suspect it is not portable (due to socket FDs not being pollable by GLib without wrapping them in a GIOChannelbasically). If someone is curious, it is here: https://github.com/jhenstridge/asyncio-glib/pull/10

[benzea]

@lazka already asked earlier, a clarification of the gbulb license would be helpful. Without that one might need to start from scratch when trying to integrate it into pygobject.

EDIT: Ohh, looks like there is an Apache license file now. But it looks to me like that is not compatible with LGPL-2.1+.

[begnac]

Somewhat tangential: I used to use gbulb, but, well, unmaintained and all, so I set out to redo from scratch, with a lazy man's approach of modifying existing loop implementations as little as necessary. It's very basic, but works for my application (an MPD client, mixing Gtk user interface and async socket communication with the mpd server). And can theoretically be improved for other use cases. You can find it here.

[freakboy3742]

FYI folks; @nhoad has just transferred ownership of this project to me.

@lazka If merging this into PyGObject is still an option, I'm open to helping out (and I agree that PyGObject is a natural place for this sort of code to live).

[benzea]

@freakboy3742, in case you have not seen it. Some time ago I worked on adding asyncio support for pygobject itself (i.e. Gio async routines). See https://gitlab.gnome.org/GNOME/pygobject/-/merge_requests/158

My plan was to hack up a thin asyncio.SelectorEventLoop wrapper that is good enough for Linux ( asyncio.py is my WIP for that; sorry, I don't think that version actually works).

That said, GBulb seem really neat feature wise in other regards. So maybe that is the better solution in the end, especially if someone is interested in maintaining it.

Anyway, if you are interested, maybe we should sync up a bit on what we can do. I should be able to spend some time on it, feel free to ping me on IRC (my nick is benzea on various networks, best is probably on GIMPnet in #python). Note that I am not a maintainer though, and so far it seemed to me that the interest in merging all this is pretty low.

[chrysn]

After the originally referenced issue has been migrated to pygobject's new issue 146, with the pygobject activity focusing on MR 189.

[freakboy3742]

@chrysn Thanks for the heads up. FWIW, I'd vastly prefer to use functionality baked into PyGObject. I maintain this package out of necessity, not out of any deep interest in maintaining asyncio support in GTK. If PyGObject gains enough baked-in support for asyncio loop integration to meet my needs for Toga, I'd deprecate this project in a heartbeat.

[benzea]

This issue should be obsolete with MR 189 merged as an experimental feature.

Please note the following that this feature is experimental, so try it but don't rely on it just yet. More specifically:

• The eventloop itself is hopefully in a good state.
• Gio async function integration still requires multiple changes, which might result in breakage.

[freakboy3742]

Closing on the basis that PyGObject now has native asyncio support in a public release.