Fixed buffers to support ReadFixed and WriteFixed ops

[songzhi]

Provides fixed buffers from a pre-registered buffer pool, to support ReadFixed and WriteFixed ops. The APIs monoio should provide are important and should be discussed.

There are several related works. glommio uses a per-reactor buddy-system allocator to allocate buffers for writes to their DmaFile. The DmaFile needs the buffer and the position be properly aligned for Direct I/O. But monoio doesn't suppor Direct I/O now. In the tokio-uring's pending PR, the APIs are as follows:

struct FixedBuf {}
struct BufRegistry {}

impl BufRegistry {
    pub fn new(bufs: impl IntoIterator<Item = Vec<u8>>) -> Self {}
    pub fn register(&self) -> io::Result<()> {}
    pub fn unregister(&self) -> io::Result<()> {}
    pub fn check_out(&self, index: usize) -> Option<FixedBuf> {}
}

impl File {
    pub async fn read_fixed_at(
        &self,
        buf: Slice<FixedBuf>,
        pos: u64,
    ) -> crate::BufResult<usize, Slice<FixedBuf>> {}

    pub async fn write_fixed_at(
        &self,
        buf: Slice<FixedBuf>,
        pos: u64,
    ) -> crate::BufResult<usize, Slice<FixedBuf>> {}
}

Should monoio provide extra read_fixed_at and write_fixed_at methods which only accept FixedBuf as parameter, or extend IoBuf/IoBufMut to provide a is_registered/is_fixed hint to allow calling Op<ReadFixed/WriteFixed> internally in read_at/write_at?

How the fixed buffer allocator/registry should be like? I prefer to provide a allocator like glommio, and provide a register buffers method, but let users manage these fixed buffers, monoio only take and return their ownerships.

I'm glad to implement this feature as long as the discussion ends up with a result.

[ihciah]

So you mean:

1. Expose a function to RuntimeBuilder to register buffers.
2. Add registered_buffer_id(&self) -> Option<u16> to IoBuf and IoBufMut(The function sign is just for demo).

I don't know if there are some limitations of io-uring, like if it is allowd to register the same buffers to multiple rings. Also, this will bring the limitation that we cannot have multiple rings(like glommio) in a single Runtime instance.

The buffer manager is another thing I think. We can let users do it by themselves or provide an impl for them.

[songzhi]

Just checked glommio's source code, they registered the same buffer to all rings. But the buffer index in different rings maybe different, so this is a problem when exposing the register buffers function to users.

https://github.com/DataDog/glommio/blob/16d9166c4753e93ecd886237d6dbd48eb8ab3e5c/glommio/src/sys/uring.rs#L1200-L1229