Support for Zero Copy via the SENDZC family of operations

DataDog / glommio

Glommio is a thread-per-core crate that makes writing highly parallel asynchronous applications in a thread-per-core architecture easier for rustaceans.

Other

3.1k stars 165 forks source link

Support for Zero Copy via the SENDZC family of operations #561

Open rrichardson opened 2 years ago

rrichardson commented 2 years ago

io_ring supports the registration and re-use of buffers via the SENDZC family of operations:

See: https://lwn.net/ml/linux-kernel/cover.1640029579.git.asml.silence@gmail.com/ https://lwn.net/ml/linux-kernel/af0d6b3e-85d0-9b61-4acd-5eef1375f934@gmail.com/ https://lwn.net/Articles/879724/

I'm still learning Glommio, but it seems like it'd be a fair amount of work to support:

The creation/management of numbered contexts. Presumably these would correspond 1:1 with executor threads, but maybe not.
An API to allocate buffers and hand them to the context.
Additional options for sockets to be able to leverage the zero-copy buffers.
Functionality to "collect" the free-buffers and present to the user for re-use.
Probably lots of other stuff

If there is interest in such features, I can take a stab at implementing them. This seems like it'd be very beneficial for my use-case.

glommer commented 2 years ago

@rrichardson yes, this feature is relatively recent on io_uring so we haven't considered it yet. But it's a very good addition!

Another thing that would be fantastic would be something like this: https://git.kernel.dk/cgit/linux-block/commit/?h=io_uring-flags2&id=ef2c2c975a921046e081a6aa29732c5c7b09cdb2

I talked to Jens about it on twitter a while ago, but I don't know if that is merged yet.

For networking in general, the biggest hindrance to maximum performance is the fact that we're following the standard rust traits, that kind of force a copy. I think we'd have to have specialized APIs to really take advantage of it.

rrichardson commented 2 years ago

Another benefit of not needing Send/Sync is that it should be easier to manage && data strutures.

We will probably not escape some unsafe while convincing rustc about the ownership of these buffers that are co-owned by io_uring.

rrichardson commented 2 years ago

Maybe an ExecutorArena data structure would be the ticket.

bryandmc commented 2 years ago

Hey @rrichardson - I definitely want to implement this at some point. That said, it is VERY new and there might be a few other features that we want to implement before this.. I am trying to consider adding them based on compatible version so that we aren't making a feature that only 5 guys running 'master' can even run..

We have a decent amount of experience sharing buffers with io_uring so we are definitely aware of what it will require. It's really just a matter of finding the time and someone motivated to do it. I want to do a few of these things but I haven't had the time since the birth of my first child..

We always appreciate patches from first time contributors! just throwing that out there ;-p .. anyways thanks for mentioning this and I'll keep the ticket open until we have some more traction on this specific feature...

glommer commented 2 years ago

@bryandmc congrats on your first kid!! Totally unrelated to this ticket of course, but my third is coming in a couple of months. The second was already a crazy ride, and the third a total surprise. So definitely understand you!