search

cornelisnetworks / opa-psm2

Other
37 stars 29 forks source link

Obtain source rank from psm2_mq_status ? #10

Closed danghvu closed 7 years ago

danghvu commented 7 years ago

Section 4.3.2.1 specifies a msg_source in the field of psm2_mq_status.

However I do not find this field in the master branch code: https://github.com/01org/opa-psm2/blob/master/psm2_mq.h#L380

What is the proper way to obtain the source rank?

rwmcguir commented 7 years ago

It appears the PDF file is incorrect, digging through the history (far older that what git hub contains) I don't see any reference to this member variable. I will have the PDF corrected.

Let me get back with you on obtaining the source rank.

rwmcguir commented 7 years ago

Assuming you are using PSM2 natively and not through MPI. In this context rank is purely a MPI defined notion. PSM2 does not need rank information to operate or distinguish messages. If you are using MPI in conjunction with PSM2 you will need to exchange rank information yourself and build a lookup table using epaddrs. You can also try to extract it from the tag. PSM2 doesn't track rank information internally other than for attempting load balancing for HW resources during job startup.

The preferred method is to switch to the API using the psm2_mq_status2 structure. This has the remote epaddr as part of the status field.

/* Remote peer's epaddr / psm2_epaddr_t msg_peer;

It is also recommend if you look into the psm2_mq.h file for examples. A good number of refernces on how MPI uses tags and rank id is present. The new tags are larger in psm2 vs psm1 so more information can be packed in, so any place you see data structures or APIs that are duplicated except with a 2 in them, that is expansion for PSM2 using the 96 bit tags (vs 48 bits in TrueScale PSM1).

I hope this assists you.

danghvu commented 7 years ago

Thanks, turns out it's easier to use the tag interface to transfer the source id. Update to the pdf will be great.