C/Rust libraries and Trio introp documentation

[thedrow]

Suppose I have a C library such as libpq or hiredis which exposes an API for async communication with their servers. I'd like to create a python client that supports trio as the event loop but performs the socket operations on it's own. The asyncio version is quite clear, albeit a bit lengthy. I have not seen an example of a C extension that cooperates with the trio event loop as of yet.

I believe we should document how to do it correctly. What do you think?

[codypiersall]

A nice little explanation/tutorial for this would be pretty helpful. Is this issue just asking for docs, or for pointers about how to actually implement this for the two libraries you mentioned?

[thedrow]

This issue is about documentation. I think that a simple echo client/server would suffice.

If you have concrete examples of client libraries written in C or Rust working with trio, I'd be happy to examine them.

[oremanj]

I'm not familiar with the particular libraries you mentioned, but if the library gives you file descriptors and asks for callbacks when they become readable/writable, you might be looking for trio.hazmat.wait_readable and wait_writable?

[codypiersall]

The library pynng, which is a Python wrapper for nng using cffi, does this. @njsmith and @tgoodlet helped me with the implementation a lot. The issue that tracked the ideas was https://github.com/codypiersall/pynng/issues/4 and the commit that added support was https://github.com/codypiersall/pynng/commit/ec99bda32f753f4883a0360d53038990d3fe1f86.

I think the integration was more-or-less straightforward, with the exception for me being handling of cancellations (abort_fn in @njsmith's comment) ; maybe I just didn't read the right docs, but I don't really get the interaction that's going on there with Trio's event loop.

Integrating with nng was pretty straightforward because nng's aio API takes a callback, which is called when the aio completes, so I was able to easily call token.run_sync_soon(trio.hazmat.reschedule, task) from the callback. For libpq I couldn't find something that notifies you when an operation is complete; I only found the method of polling occasionally to see if an aio completes. I probably missed something though since I'm completely unfamiliar with that library.

[oremanj]

I just took a look at hiredis, and based on https://github.com/redis/hiredis/blob/master/adapters/libevent.h it seems that the library wants certain functions to be called when a particular file descriptor becomes readable or writable, plus it wants callbacks that it can invoke to enable/disable readability/writability notifications. This is a common approach used by C libraries, but is very different than the approach taken by nng, which does its I/O in a thread and hides more of the plumbing from the user.

The readability/writability callbacks approach is a slightly awkward fit for Trio only because Trio is so adamant about not using callbacks in its public APIs. :-) But it's not impossible by any means. Very roughly, you'd probably want to create Trio tasks that perform a loop of [call wait_readable or wait_writable, tell hiredis that the thing happened], and spawn each task when hiredis starts being interested in that event / cancel it when it's no longer interested. (Or the tasks could stay running continuously, and just switch between blocking on wait_readable and blocking on a trio.Event. Or whatever.)

I agree this would be a good example to have, since it's a common pattern and there are some tricky details. Unfortunately it doesn't appear that the Python hiredis bindings https://github.com/redis/hiredis-py expose hiredis's I/O logic, so we'd need to improve the bindings or use cffi. I might find time to work this out at some point, but probably not soon.

[oremanj]

It would be nice to have an example wrapping the c-ares asynchronous DNS resolver. There's already a Python package for it (pycares) which has decent async primitives support, although it's very callback-based.