Migration to `trio`

[ralexstokes]

This issue serves as a starting point to discuss migrating from asyncio to trio.

trio builds upon the insight gained from use of asyncio across the Python ecosystem; in particular, it provides support for a structured approach to concurrency that enables the correct construction of large asynchronous programs.

A simple migration is to just replace use of asyncio with the use of trio.

A more complex, but possibly more useful path is to consider how we can abstract out the use of IO in py-libp2p so that a client of the library can supply their own IO. Refer to this https://sans-io.readthedocs.io/ for more information on what I'm thinking here.

[ralexstokes]

Also worth considering using this: https://github.com/python-trio/trio-asyncio

[stuckinaboot]

Re: Migration to trio- In my opinion, the top priority right now should be completing #169 since when that issue is completed, py-libp2p will be a fully interoperable networking stack of libp2p (which will be a major milestone for libp2p in general and for all projects that intend to use py-libp2p). Once that is done, optimizations could/should be done but until then I feel they may complicate progress towards the completion of #169.

Re: trio- I think this could be pretty useful. With regards to the more complex path, I think that could be very useful as long as it can be done in a way that doesn't greatly complicate the creation of py-libp2p nodes. Part of the beauty of py-libp2p right now is how simple it is to both create nodes and add complex communication between nodes (e.g. pubsub), and so if more flexibility could be added without compromising ease-of-use (possibly by using asyncio or trio as a default IO library if the client does not supply one of their own) that could be great.

[pipermerriam]

I'd recommend starting with trio-asyncio, verifying that the performance is still in an acceptable range and then doing the rest piece-by-piece.

[ShadowJonathan]

Would it be possible to make py-libp2p async-library-neutral? So clients and programs can start the event loop with their own implimentations (asyncio, trio, or other), and libp2p can adjust to whatever they choose at runtime?

[ralexstokes]

it is definitely possible... the transparent route is to use something like anyio https://pypi.org/project/anyio/ but i think there is some concern around performance. we would need more research/benchmarking to better understand the tradeoffs here....

another route is to factor the IO out of the library entirely and then we can provide "driver" libraries that consume IO events in their own way -- this is the "sans IO" approach linked here: https://github.com/libp2p/py-libp2p/issues/174#issue-458267139

edit: i think the current aim of the maintainers is to go all in on trio, although this isn't a final position. it could also likely be the case that we switch to trio now and then down the road do a bigger refactoring a la sans io

[ShadowJonathan]

Given that libp2p is all about modularity, and that a library's async/io implimentation can make or break it's integration in other projects, I indeed also vote for factoring out the io in that final release

[pipermerriam]

@ShadowJonathan we are :+1: to going this direction, but we may take a bit of a pragmatic first step towards getting it working with trio first since full io abstraction is a more complex piece of work and is likely to take more thought and effort. At present this library is primarily used in trinity codebase and the two are being developed together.

I suspect this is something that we'll entertain focusing on in the 6-12 month timeline.