search

erpc-io / eRPC

Efficient RPCs for datacenter networks
Other
851 stars 138 forks source link

When does transport layer make data visible to app, and in what granularity #19

Closed lastweek closed 5 years ago

lastweek commented 5 years ago

Hi Anuj,

This issue maybe not eRPC specific, but more from RDMA's angle. I just thought you are the right person to ask.

Both eRPC and RDMA break big messages into small packets, and use go-back-N to handle packet loss. My concern here lies in: when does the transport layer make data visible to application, and in what granularity.

My understanding about eRPC: eRPC won't export the data to application until all packets of a message has been received. In another words, eRPC does not allow application to see a partial received message.

While RDMA (e.g., WRITE) is different: the RNIC will just write each packet directly into final DRAM location. That means if there is an application (e.g., FARM) polling DRAM, the application will see the new data the moment the first packet is received, and in packet granularity. One reason of this, is because RNIC does not have on-chip memory to hold out-of-order packets. The IRN paper sort of confirmed my understanding of this.

Do you think my understanding is correct? Thank you in advance.

anujkaliaiitd commented 5 years ago

Hi Yizhou,

Your understanding of eRPC is correct.

For RDMA writes, the remote machine might see only a few bytes of a large packet. On current Mellanox NICs, the bytes and packets materialize in left-to-right order, but this guarantee is not provided by the spec, and it's likely to be violated in future NICs.

lastweek commented 5 years ago

Right. I did not know this until read about IRN, I always thought left-to-right order is a standard feature of RDMA operations. Thank you Anuj. :-)