LCI_putb

[omor1]

PaRSEC needs LCI_putb support

[danghvu]

Please expand these issues a bit with why you need it, typical usage, what kind of completion (local, remote) etc..

[omor1]

For PaRSEC's send_active_message communication engine method, my plan was to send buffered data via putb and get it out via a handler or completion queue. The tag determines what callback is called.

[omor1]

LCI_putb or LCI_putmd needs to be implemented. The latter has a declaration, but no definition I believe. The former would additionally the LCI buffers concept. I can work with either; whichever is simplest to get running would be best.

[omor1]

I can take a gander at being how to implement this. @danghvu maybe we can meet at some point to give me some pointers.

[danghvu]

Happy to give some pointers

I would start by looking at how puts is implemented in the server code, i think the closest is probably lc_server_putm

Are you on infiniband or psm?

[omor1]

I'll run either on our XSEDE allocation (Stampede2 and Comet) or on Bebop at LCRC, more likely probably the former rather than the latter. Focussing on psm2 first is fine then, since that's used on both Stampede2 and Bebop; IB is used on Comet. So I guess whichever is easier.

How is dynamic allocation send/put implemented? All I see in the current implementation is when the destination is known. My guess is we'd basically do a send, but steal a bit in the tag to indicate that the destination buffer should be allocated remotely? Or something like that.

[omor1]

Right, PSM2 doesn't actually have RDMA, so one-sided is done via send/recv. Ugh, this means that we have multiple levels of faking one-sided in LCI/PaRSEC, which is a shame.

So for putmd the way to implement it over PSM2 would be to add a new message type that indicates that the data should be allocated by LCI and use the tag to send the size. When the receiver processes the size message, it allocates an array of the specified size and posts the send for it, which is matched with the second send containing the data.

I think that sounds right?

[snirmarc]

psm2 is probably dying. Intel support Libfabric. So I would focus on IB and Libfabric. Check with Vu about the status of the Libfabric port.

On Dec 9, 2019, at 3:33 PM, Omri Mor notifications@github.com wrote:

Right, PSM2 doesn't actually have RDMA, so one-sided is done via send/recv. Ugh, this means that we have multiple levels of faking one-sided in LCI/PaRSEC, which is a shame.

So for putmd the way to implement it over PSM2 would be to add a new message type that indicates that the data should be allocated by LCI and use the tag to send the size. When the receiver processes the size message, it allocates an array of the specified size and posts the send for it, which is matched with the second send containing the data.

I think that sounds right?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/uiuc-hpc/LC/issues/5?email_source=notifications&email_token=ABLZHITCAIHT3QFUAG4RXFLQX22SXA5CNFSM4HFCNOA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGKY7OQ#issuecomment-563449786, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLZHIXHWSS7EHQ6HIVV26TQX22SXANCNFSM4HFCNOAQ.

[omor1]

psm2 is probably dying

Probably correct, seeing as Omni-Path 2 is dead as far as we know.

status of the Libfabric port

Based on the unimplemented functions, there's probably some work needs to be done to get it in shape...

[omor1]

For OFI the implementation would probably use the fi_tagged interface with either Buffered Tagged Receives (for putb) or Variable Length Tagged Messages (for putmd); these were introduced in OFI v1.7.x (current is v1.9.x). Not sure what OFI version we're targeting right now.

[danghvu]

I will push some stuff for psm2 putb today. Libabric on psm2 is a bit buggy since it gives a bunch of tag bit for intetnal use.

I think for libfabric we want to test on infiniband verb NIC

On Mon, Dec 9, 2019 at 3:55 PM Omri Mor notifications@github.com wrote:

For OFI the implementation would probably be using the fi_tagged interface with either Buffered Tagged Receives (for putb) or Variable Length Tagged Messages (for putmd); these were introduced in OFI v1.7.x (current is v1.9.x). Not sure what OFI version we're targeting right now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/uiuc-hpc/LC/issues/5?email_source=notifications&email_token=AAIZNSXA7L7NMJEE2XI5U2TQX3LFZA5CNFSM4HFCNOA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGLD5KQ#issuecomment-563494570, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIZNSVTEBCCRNTMZ6T5LBTQX3LFZANCNFSM4HFCNOAQ .

[omor1]

Reading the fi_psm2 manpage it seems that there's a couple knobs that can be used to change the tag layout? FI_PSM2_TAG_LAYOUT set to tag60 only leaves 60 bits (out of 64 in Libfabric) to to application; tag64 allows all 64, but then has remote CQ data limitations. There's also FI_PSM2_TAGGED_RMA which uses a bit to implement RMA over tagged messages rather than active messages, but it's unclear if this is stealing a bit from those available to the application or whether it's one of the 4 bits (out of 96 total in PSM2) used for the internal flags.

[omor1]

My last impression was that the verbs provider didn't have good performance/support of OFI/libfabric, though that could have changed. There used to be (until recently) the Mellanox provider that used UCX, but that wasn't being maintained and was removed.

[danghvu]

Yes that is why we shouldnt use it and add hack to our code because of remote knowledge. Ibverb may be slow but it seems to implement the correct protocol without hack inside libfabric. If we want performance the native layer is the only way.

On Tue, Dec 10, 2019 at 2:24 PM Omri Mor notifications@github.com wrote:

Reading the fi_psm2 manpage it seems that there's a couple knobs that can be used to chang the tag layout? FI_PSM2_TAG_LAYOUT set to tag60 only leaves 60 bits (out of 64 in Libfabric) to to application; tag64 allows all 64, but then has remote CQ data limitations. There's also FI_PSM2_TAGGED_RMA which uses a bit to implement RMA over tagged messages rather than active messages, but it's unclear if this is stealing a bit from those available to the application or whether it's one of the 4 bits (out of 96 total in PSM2) used for the internal flags.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/uiuc-hpc/LC/issues/5?email_source=notifications&email_token=AAIZNSSTGFLAKJ2CZKZBB5LQYAJIBA5CNFSM4HFCNOA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGRFNGQ#issuecomment-564287130, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIZNSRVPCATSOIOFTZRXTLQYAJIBANCNFSM4HFCNOAQ .

[danghvu]

Putb is done for PSM2