search

charmplusplus / charm

The Charm++ parallel programming system. Visit https://charmplusplus.org/ for more information.
Apache License 2.0
203 stars 49 forks source link

Zero-copy send support for the netlrts machine layer #1459

Open PhilMiller opened 7 years ago

PhilMiller commented 7 years ago

Original issue: https://charm.cs.illinois.edu/redmine/issues/1459

In the netlrts machine layer, it's pretty easy to stream data from an arbitrary address to a remote recipient on request. We can use this to implement zero-copy sends for the memory footprint reduction, and to take copying time overhead off the worker threads and onto the comm thread.

nitbhat commented 5 years ago

Original date: 2017-05-09 00:18:41

The current netlrts layer (UDP) in machine-eth.c sends a Datagram header with every packet it sends. For every packet, it creates a header at (char *ptr - DGRAM_HEADER_SIZE) and sends it over. The receiver uses the packet information to assemble packets together.

I am guessing that we’re able to use the DGRAM_HEADER_SIZE bytes before the packet ptr because of two reasons:

  1. We know that the previous packet has been delivered? (using acks)
  2. It is the Charm++ owned copied buffer and free from user intervention and will be freed after the message send.

But, if we’re to send a user buffer using this scheme, how do we send a header with every packet? (as we can’t touch it to send the header info at a negative offset)

PhilMiller commented 5 years ago

Original date: 2017-05-09 00:28:05

We could do the packetization in a set aside buffer that we copy the user's data through as we send it. The key is to tightly constrain the size of that buffer, making it just large enough to get full network bandwidth. If the packet injection is synchronous, then that buffer only needs to hold 1 packet.

ericjbohm commented 5 years ago

What is the current level of support for zero copy in netlrts?

nitbhat commented 5 years ago

There hasn't been any work to support Zcpy in netlrts. It currently uses the copy based generic layer.