MPI_MAX_FROM_GROUP_TAG Issues

[jdinan]

Problems

1. There is no minimum value specified for MPI_MAX_FROM_GROUP_STRINGTAG. For example MPI_MAX_OBJECT_NAME is specified to have a value of at least 64. Not having a minimum length for this string may make it hard for applications to specify unique strings. E.g. if I end up needing to truncate my string it may no longer be unique. Versus planning ahead for uniqueness within the minimum string length.

2. We have both MPI_MAX_FROM_GROUP_STRINGTAG and MPI_MAX_FROM_GROUP_TAG.

Proposal

Two possible resolutions were discussed on 12/8:

1. Remove the constants.
2. Specify that MPI_MAX_FROM_GROUP_TAG be at least 64.

Forum seems to prefer option 1.

Changes to the Text

For option 1:

• Delete the constants.

For option 2:

• Rename MPI_MAX_FROM_GROUP_STRINGTAG to MPI_MAX_FROM_GROUP_TAG.
• Add the following minimum (c.f. usage in Example 11.8 for guidance on minimum length):

  MPI_MAX_FROM_GROUP_TAG must have a value of at least 64.

Impact on Implementations

New constant, best to make this change now before it affects implementations.

Impact on Users

Makes the API easier to use. These APIs are on the critical path for all users of the Sessions API.

References

MPI 4.0 RC pages 355 and 376. Example 11.8.

387 Merged into this issue.

[dholmes-epcc-ed-ac-uk]

There is a table in Appendix A (§A.1.1, p848, title "Maximum Sizes for Strings") that lists all of the constants that specify maximum string lengths. Only 20% of these constants (MPI_MAX_OBJECT_NAME and MPI_MAX_DATAREP_STRING) have a minimum value defined (both 64).

§7.8, p376, line 26.5:

MPI_MAX_OBJECT_NAME must have a value of at least 64.

§14.5.3, p692, line 22:

MPI_MAX_DATAREP_STRING must have a value of at least 64.

I support this change - is 64 enough for this purpose (I think yes)?

[jdinan]

In this particular case, I think having a minimum will help users quite a bit, since creating a unique string that fits in an unknown amount of space is much harder than creating a unique string that fits in a small, but known amount of space.

[jdinan]

C.f. String usage in example 11.8:

rc = MPI_Comm_create_from_group(wgroup,
        "org.mpi-forum.mpi-v4_0.example-ex10_8",
        MPI_INFO_NULL,
        MPI_ERRORS_RETURN,
        &lib_comm);

[jdinan]

Resolution proposed in 12/8 meeting is to remove the constants.

[jdinan]

Capturing a few notes from chat discussion:

from Bill Gropp to Everyone: 10:11 AM As an implementor, I'm not worried about strings that aren't on the performance critical path and don't require long-term storage (thus don't threaten the memory scalability). Using chat to get a word in edgewise.

from Bill Gropp to Everyone: 10:15 AM MAX lengths were typically used for OUTPUT strings, since Fortran77 could not return an arbitray length string.

from Dan Holmes EPCC to Everyone: 10:19 AM Bill has a point - the PSET procedure mentioned by Howard includes an output string, this is an input string instead

[hppritcha]

I looked into this some for our sessions prototype. The prototype relies on an external package - PMIx - to implement the core functionality of MPI_Comm_create_from_group. In particular, the prototype relies on PMIx_Group_construct. And indeed, the tag argument supplied to this PMIx function does indeed have a max length. So at least for this prototype, its important to have a max len for the tag supplied by the application to MPI_Comm_create_from_group and related intercommunicator constructor.

[dholmes-epcc-ed-ac-uk]

What is the current value for the maximum length in PMIx? 64 seems arbitrarily small to me.

[besnardjb]

A quick look at the implementation of PMIx_Group_construct here points to PMIX_MAX_NSLEN valued 255 and as per the upcoming standard 4.0 (p. 32 of the PDF) it should at least be 63.

[dholmes-epcc-ed-ac-uk]

Could we ask the PMIx folk: 1) is there a good reason for choosing 63 for the standardised limit? 2) is there a good reason for choosing 255 for the implementation value? 3) is there some underlying restriction that prevents these numbers being higher?

[jjhursey]

From what I remember of the discussion around PMIX_MAX_NSLEN (defined in pmix v4.0rc5 Section 3.1 lines 15-18):

• Some context:
  • The PMIx namespace string is a piece of information circulated with nearly every PMIx activity since it defines the "who" in the message - either the target or destination of the action. Processes are identified in the system by their namespace and rank.
  • Since the namespace is circulated so often, the length of the string could add overhead to messages passed between RMs (though that is RM implementation-specific), and may impact the on-node storage for the key-value store (again depending on the PMIx implementation).
  • For PMIx_Group_construct the grp string is limited to PMIX_MAX_NSLEN because it is used to identify the set of processes in the group - similar to how the namespace is used more broadly.
• In the standard we needed to define a minimum value for the client and/or server/RM to target when thinking about portable string values.
  • We chose 63 instead of 64 to leave space for the NUL character.
  • We chose 63 over 255 to allow for PMIx implementations to explore optimizations around the namespace per above.
  • IIRC, we chose 255 for OpenPMIx for historical reasons - earlier versions of OpenPMIx had this limit and it gives plenty of room.
  • I don't remember why we chose 63 over any other value - larger or smaller.

IIRC @abouteiller and @rhc54 were active in that discussion so might have other points to add or clarifications.

[rhc54]

Digging back into my deep, dark memory spaces...

Originally, we didn't have a minimum length - we only specified the maximum. We picked 256 as the size of the namespace array as it fit on a memory boundary and we wanted it large enough to at least hold an Internet URI (since prior OMPI members had used such things as their "namespace"-like identifiers). A size of 64 wasn't adequate for that purpose.

At a later point in time, there was interest in simplifying the way OMPI integrated with PMIx. OMPI uses an integer jobID in place of the string namespace, and so we were having to convert back-and-forth fairly often. The proposed optimization replaced the integer jobID with the PMIx namespace string. However, as we scaled the system to ever larger sizes, the memory footprint of that fixed 256-char array began to become an issue.

This is when we added a minimum size to the Standard. In nearly every system using PMIx, we never saw a need to use more than a few characters in the namespace - mostly because the namespace was treated as a string representation of an integer jobID. In those cases where the namespace was some kind of URI (e.g., for a tool started on the cmd line), we still didn't see lengths greater than about 30-40 characters.

Hence, we chose 64 characters as the minimum array length - again, chosen to align with memory boundaries. If you subtract the space required for a NULL terminator on the string, you get our current values for min and max namespace length.

We could have switched to an unconstrained length, just making the namespace arguments into char*. We considered that option but chose not to pursue it as it resulted in a lot of malloc operations. Although PMIx should never be in a critical path, we still wanted to conform to reasonable performance coding methods. So we stuck with the fixed array and defined min/max lengths.

HTH

[dholmes-epcc-ed-ac-uk]

We should specify "shall have a value at least 63 characters" then - meaning 64 bytes buffer space (including NULL char) in C or 63 bytes buffer space in Fortran.

[hppritcha]

related to https://github.com/mpi-forum/mpi-standard/pull/400