ch4: add MPIDI_NM_am_tag_send/recv

[hzhou]

Pull Request Description

In mpidig RNDV, if the netmod supports tag -- query via MPIDI_NM_am_can_do_tag() -- add MPIDI_NM_amtag{send,recv} to support direct send/recv. This not only avoids the extra copy in UCX or extra hdr message and additional RDMA read in OFI am_send_long, it also supports native GPU RDMA direct when the netmod supports it.

This is how it works -

• sender sends RTS
• receiver posts MPIDI_NM_am_tag_recv and reply CTS
• sender do MPIDI_NM_am_tag_send

Strength

• Simple - avoid additional transport-layer protocols
• Flexible - ch4-layer unified am protocols
• Efficient - native matching + native RDMA

Comparisons

• Existing OFI

  

• Existing UCX

  

Follow-up

• OFI native path transition to AM protocols

[skip warnings]

Author Checklist

• [x] Provide Description Particularly focus on why, not what. Reference background, issues, test failures, xfail entries, etc.
• [x] Commits Follow Good Practice Commits are self-contained and do not do two things at once. Commit message is of the form: module: short description Commit message explains what's in the commit.
• [x] Passes All Tests Whitespace checker. Warnings test. Additional tests via comments.
• [x] Contribution Agreement For non-Argonne authors, check contribution agreement. If necessary, request an explicit comment from your companies PR approval manager.

[hzhou]

test:mpich/ch4/most ✔️ test:mpich/ch4/ofi/more - am-only has some issue

[hzhou]

test:mpich/ch4/ofi/more ✔️