Reconsider API design

[Noah-Kennedy]

Currently, our API just adapts the existing tokio APIs to be completion-based. While this makes the library easier to use, it raises a number of problems for the maintenance of these APIs.

io_uring has a large number of useful flags which can be applied to operations, and more continue to be added. We cannot effectively make use of many of these flags (e.g event linking flags, buffer preallocation flags, etc.) without adding more and more functions with larger signature or issuing frequent breaking changes to said functions as new useful flags come along, which might continue to occur for some time.

I propose that we move to a builder API for operations. A builder pattern would allow operations to be constructed, with decisions like buffer strategy selection and operation linking made before the operation is dispatched to yield a future. This would offer a tremendous benefit especially to power users of io_uring, who may be trying to take advantage of various features of the API to improve performance or reliability.

[Ralith]

In Quinn, we currently bypass tokio 1.0 I/O APIs entirely because they can't possibly express the eclectic collection of options and ancillary data we rely on (e.g. IPV6_TCLASS, IPV6_PKTINFO, their IPv4 analogs, UDP_GRO, UDP_SEGMENT, etc). Something more scaleable would be nice! These interfaces can be pretty diverse, though, so accounting for every possibility may be difficult.

[Noah-Kennedy]

@Ralith I know the feelling quite well, and I was actually thinking about some similar issues we've had at work related to the tokio APIs when I came up with this proposal.

I also think that, as a cop out, we could also expose a function which allows "raw" events to be submitted.

[mzabaluev]

The explosion of methods exemplified by #184 make me support the idea of builder APIs.

Something like this would be nice:

let n = file.build_write()
    .fixed(buf)
    .at(pos)
    .start()
    .await?;

It may be even possible to eliminate Slice and provide a way to multi-slice a buffer for compound operations:

file.build_write_all()
    .buf(v)
    .multi()
    .slice_at(..5, 0)
    .slice_at(7..10, 5)
    .start()
    .await?;

[Noah-Kennedy]

I'm currently working on this but it's blocked behind another PR which I need to push through first. Hopefully I get the "refactor the runtime to use internal handles" PR up tonight as planned, so that I can resume working on this.

[Noah-Kennedy]

An update on this: #244 kicks off the start of this, initially for only oneshot operations.