Support UDT

[johanmickos]

We want to enable UDT in the transport layer. UDT comes as a C++ library with a small API surface, and gluing it to be supported in Rust is surprisingly simple (see this repo, for example).

Theoretical Approach

1. Write or find C-bindings to glue Rust to the C++ API
2. Wrap the epoll-based API calls with Mio support (through the Evented interface
3. Implement read/write over the provided UDT sockets/streams
4. Let the implementations from Tokio, Futures, and Mio to do their magic and have full asynchronous support for UDT in Tokio land

Problems

Mio allows two ways to register a pollable resource through its Evented interface: (1) user handles, driven in user-space by hand-written "readiness" updates; and (2) system file handles such as sockets that can be monitored by the system selector, thus delegating readiness updates to the operating system.

The epoll-like API exposed by UDT operates on logical UDT sockets rather than system-specific file descriptors. This means that there's no system file handle providing epoll-like capabilities that Mio's Evented can use. For Mio integration, we're left with option #2: writing a dedicated user-space poller which drives the readiness state of UDT sockets (presumably from a separate thread). In other words, re-writing the Poll functionality found in Mio to work on "something that looks like epoll but isn't necessarily the system selector."

There is a problem with the user-space Evented types, though: they come with very few guarantees. See the Mio quote below:

There is no guarantee that readiness establishes any sort of memory ordering. Any concurrent data access must be synchronized using another strategy. There is also no guarantee as to when the readiness event will be delivered to poll. A best attempt will be made to make the delivery in a "timely" fashion.

What about Netty?

Netty uses a very complex tree of interfaces and abstract classes. Netty also supports Java's NIO tree of interfaces and abstract classes. Within NIO, we find selectors which multiplex channels. The API exposed by the UDT library fits quite nicely into NIO's selectors, and thus Netty's single-threaded NioEventLoop can drive the polling on the UDT resources and pipeline them through the rest of Netty's features.

Potential Workarounds

1. Stick to Mio and Tokio for UDT and write a custom poller to drive the UDT library and the connections registered in it, forwarding the epoll-like events from UDT to Mio's readiness events. This approach can then plug into the futures-oriented pipelines of Tokio which is a big plus, but comes at the drawbacks of being time-consuming to write and having loose guarantees provided by Mio.
2. Drive UDT connections in a separate UDT-dedicated thread, funneling recv/send messages between the dedicated thread and the dispatcher using channels.
3. Rewrite the UDT library in Rust with Mio support from the core, allowing Tokio and Mio to drive the underlying UDP sockets, and implementing the protocol logic using futures.
4. Other ideas?

[Bathtor]

There still does not appear to be any change to the state of this in the Rust ecosystem. The only native rewrite attempt appears to be abandoned.

[gpicron]

What about QUIC ? It is loosely inherited from UDT. 2 rust implementations of it Quiche and Quinn both interoperable with tokio/mio.

[Bathtor]

What about QUIC ? It is loosely inherited from UDT. 2 rust implementations of it Quiche and Quinn both interoperable with tokio/mio.

Just taking a quick look at the two mentioned libraries, I'm thinking this may be a pretty good option, actually. I particularly like the support unordered streams as a feature, which UDT didn't have iirc.

We'll have some internal discussions and try to get it on the roadmap. It's certainly more alive than UDT.