Get large messages sent via the ZC API to be received directly using a GET from the new home. (rather than a GET from the old home and forwarding the large message)

[PhilMiller]

Original issue: https://charm.cs.illinois.edu/redmine/issues/69

---

On systems that communicate large messages via RDMA GET operations, it takes the latency of one fewer packet across the network (4 vs 5) to convey a message to a migrated object if the sender passes a GET handle to the home PE, which then informs the recipient PE directly about where to get the message in addition to telling the sender where that object lives for future sends. Where available, pass a GET handle along with the location request to enable this.

[PhilMiller]

Original date: 2016-09-23 16:07:11

---

Nitin, Vipul: Your recent work on RDMA support probably makes this and #68 easy to implement as well.

[nitbhat]

Original date: 2019-03-20 21:41:26

---

I think Phil is trying to basically describe a “zero-copy” based transfer scheme for a large message (whose target is a migrated object) from the original object location (PE) to the new object location (PE). Is that right? I'm guessing that we currently use a copy-based scheme for forwarding the message from the original PE to the new PE?

[nitbhat]

Original date: 2019-04-03 20:05:10

---

Juan and I had a discussion about this feature on slack.

We finally understood this feature to some extent.

Let's say a chare array element 'c' migrates from PE 'y' to PE 'z'. Currently, if another chare array element 'b' on PE 'x' wants to send a large message to 'c', it sends the message to PE 'y' and then PE 'y' forwards it to PE 'z'. With this feature, chare array element 'b' on PE 'x' will not send the entire large message to PE 'y', it will only send the buffer information (GET handle) to PE 'y' and then PE 'y' will forward the buffer information (GET handle) to PE 'z'. Following that, PE 'z' will perform a GET operation to get the data directly from PE 'x'. This ends up saving the latency of one hop for the large message in addition to avoiding the memory allocation on PE 'y'.

[nitbhat]

Original date: 2019-04-03 20:49:40

---

This feature is only relevant for use cases of the Zero copy Entry Method API and not the regular API.

This API basically impacts use cases where a sender (PE 'x') sends a GET handle to the original home (PE 'y') and instead of the original home rgetting the entire message onto PE 'y' and then forwarding the large message to PE 'z' (the new home of the migrated chare), with this feature, PE 'y' just forwards the GET handle and PE 'z' performs the Rget.

[nitbhat]

Original date: 2019-04-10 19:51:41

---

Since this impacts a very small use case (where ZC API is used in conjunction with load balancing) and the impact is potentially small (only the iteration after load balancing), I am not including this in 6.10.