search

mpi-forum / mpi-forum-historic

Migration of old MPI Forum Trac Tickets to GitHub. New issues belong on mpi-forum/mpi-issues.
http://www.mpi-forum.org
2 stars 3 forks source link

MPI info key namespacing #411

Open mpiforumbot opened 8 years ago

mpiforumbot commented 8 years ago

Originally by balaji on 2014-02-09 21:38:44 -0600

Current MPI Info keys are not namespaced. This is problematic for both implementations and applications with respect to conflicting names with the MPI standard, and is similar to such conflicts in MPI function names or constants, which have an equivalent problem.

If an MPI implementation provides an info key called foo, and later the MPI standard (say MPI-5) decides to standardize this key with the name, but a different semantic, the implementation would need to be modified to remove it or update it, even if it is not necessarily compliant with the newer MPI standard. Applications cannot safely set this info key if a future MPI standard can change its meaning.

-Proposed Solution:*

The proposed solution is to namespace info keys with "mpi_", and add a statement in the standard that implementations are not allowed to create info keys in this namespace.

The proposed solution is:

  1. We reserve the "mpi_" namespace and forbid implementations from using keys in that name space.
  2. We rename existing info keys to start with "mpi_" and deprecate the older info key names.
mpiforumbot commented 8 years ago

Originally by balaji on 2014-02-09 22:46:29 -0600

Proposed changes are attached.

mpiforumbot commented 8 years ago

Originally by jhammond on 2014-02-10 05:53:55 -0600

I think it's perfectly reasonable to have to use preprocessor selection if one is using non-standard elements with MPI. It would be annoying to have to remember which info keys required mpi- and which ones didn't, based upon their age. I consider this more annoying that using the C preprocessor.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-02-10 09:47:04 -0600

The preprocessor selection in the application has to change for every version of MPI, even if nothing else changed in the MPI standard (the example above tries to explain this). This is exactly the same reason why MPI reserves the MPI namespace for functions and constants. I'm just extending it to info keys. With respect to remembering which keys require "mpi", you anyway have to do that to figure out which need to be preprocessor protected today.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-02-14 21:58:33 -0600

Uploaded a new version of the PDF based on the hybrid WG telecon discussion today.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-03-04 18:39:53 -0600

This reading passed. Forum suggested changes to wording:

  1. Rewrite the "mpi_" namespace is reserved wording.
  2. Say that deprecated keys are aliases to the new keys, so one can lookup the values with either one.
mpiforumbot commented 8 years ago

Originally by jsquyres on 2014-06-04 09:58:27 -0500

Minor nits from the reading:

There was debate about MPI_INFO_GET_NKEYS -- does the number returned include the aliases? I.e., if an implementation provides a single name "mpi_foo", does GET_NKEYS return 1 or 2? The conclusion is that the word "alias" is sufficient to differentiate the synonym from an actual key. I.e., the actual key is mpi_foo -- and that's what's counted by GET_NKEYS.

mpiforumbot commented 8 years ago

Originally by jsquyres on 2014-06-04 10:23:24 -0500

Straw vote before formal vote:

mpiforumbot commented 8 years ago

Originally by moody20 on 2014-06-04 10:29:33 -0500

mpi_appnum key name and MPI_APPNUM attribute name are now overloaded. I don't think it's a problem, but it could be confusing.

p356:9 Do we want to update text around info key names "corresponding" to spawn args, or is this still clear?

p356:37 - need to update info args listed in example

Some missing changes

no_locks

accumulate_ops

accumulate_ordering

I stopped looking after this to pay attention to more readings, but this ticket needs more work before being voted in.

mpiforumbot commented 8 years ago

Originally by moody20 on 2014-06-05 09:47:36 -0500

Replying to jsquyres:

Minor nits from the reading:

  • p365: need to say that the "mpi_" prefix is case insensitive.
  • p597 line 33: "keys" should be "key"
  • p597 line 30: change "forms" to "names"

There was debate about MPI_INFO_GET_NKEYS -- does the number returned include the aliases? I.e., if an implementation provides a single name "mpi_foo", does GET_NKEYS return 1 or 2? The conclusion is that the word "alias" is sufficient to differentiate the synonym from an actual key. I.e., the actual key is mpi_foo -- and that's what's counted by GET_NKEYS.

If we only count the new "mpi_" key names in GETNKEYS then we must only return the "mpi" keys with GET_NTHKEY, and that would break backwards compatibility in apps, for example with a function like this:

void process_my_info_in_an_inefficient_but_valid_way(MPI_Info info) {
  // ask for number of keys in info
  int nkeys;
  MPI_Info_get_nkeys(info, &nkeys);

  // iterate over every key and act on certain keys if found
  for (int i = 0; i < nkeys; i++) {
    char key[MPI_MAX_INFO_KEY];
    MPI_Info_get_nthkey(info, i, key);

    // old code only knows to look for old key names
    if (strcmp(key, "no_locks") == 0) {
      // do stuff -- this will no longer trigger if only new names are returned
    }
  }
}
mpiforumbot commented 8 years ago

Originally by balaji on 2014-06-29 13:37:37 -0500

Attached a new file to address the above comments.

In order to address Adam's comment above (comment:18), I have changed the text to say that both the old and new keys are returned by the query routines.

mpiforumbot commented 8 years ago

Originally by rsthakur on 2014-09-02 15:51:37 -0500

The new info key name mpi_file will clash with the MPI_File object name in case insensitive Fortran.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-09-02 17:37:05 -0500

Can you give an example how this will clash? The info key is a string, not a declared variable or type.

mpiforumbot commented 8 years ago

Originally by rsthakur on 2014-09-02 23:57:21 -0500

I meant in a logical sense. There will be two things in the MPI standard called mpi_file, the object and the info key. (Suppose there was an info key called mpi_send...)

mpiforumbot commented 8 years ago

Originally by balaji on 2014-09-03 00:06:15 -0500

This will not be the first such similarity. We have the MPI_AINT, MPI_OFFSET and MPI_COUNT MPI datatypes, which have similar names to their C datatypes as well. I can change the info key names, if needed, but this was not considered a problem in the MPI standard so far.

mpiforumbot commented 8 years ago

Originally by rsthakur on 2014-09-03 00:11:41 -0500

At least they stand for the same thing. The file info key in the dynamic process chapter stands for something totally different than the MPI_File object.

384:46-47 "file": Value is the name of a file in which additional information is specified. The format of the filename and internal format of the file are determined by the implementation.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-09-03 00:24:47 -0500

OK, I'll update the names to be different.

mpiforumbot commented 8 years ago

Originally by gropp on 2014-09-03 01:43:53 -0500

As this discussion goes on, it seems that a simpler approach might instead be to give a namespace to the implementation - i.e., new, non-standard names, must begin with the string mpix (or something similar). That avoids all of the issues raised above with two keys for the same value and the interpretation of changes to one or the other.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-09-03 07:20:46 -0500

I don't think we need to specify how non-standard names should be (except that they cannot start with mpi_). We should only specify that the mpi_ namespace is reserved for keys in the standard. To avoid the name conflicts, I'm planning to start all keys specified in the standard with mpi_info_.

mpiforumbot commented 8 years ago

Originally by gropp on 2014-09-07 13:50:48 -0500

I hate Trac. It discarded my original version of this note. Here's my second try, composed using a reliable program instead.

There are several orthogonal issues here. One is how to define a name space to ensure that we avoid name colisions between the standard and implementations in the future. The other is whether to change the existing names or leave them as-is. If we were starting from scratch, that second issue would not apply and the most natural and common solution is to reserve a prefix for keys defined by MPI, similar to our choice for function and object names. But that is not the situation, so it is more complicated. So here are some options, with pros and cons.

  1. MPI Standard keys have a specified prefix (e.g., "mpi_"), and all existing keys in the standard are changed to follow this rule (e.g., "file" becomes "mpi_file").\Pros: What most people would have done in the beginning.\Cons: (Nearly?)everyone agrees that for backward compatibility, the old names must continue to be accepted, at least for a while (deprecated before being removed). This leads to many difficulties, as described above. E.g., how many keys are there? If the value of "file" is changed, does that change "mpi_file"? What if "file" is given one value and "mpi_file" another? And so on.
  2. MPI implementation keys have a specified prefix (e.g., "mpix", "mpich", "openmpi_") and implementations must rename any info keys that they have defined.\Pros: Less change to the MPI standard; provides a way to see to what implementation a key belongs.\Cons: Same as for 1, but now impacting keys defined by the implementation instead of the standard.
  3. MPI Standard keys have a specified prefix (e.g., "mpi_") but existing keys are unchanged. A rationale can be added to explain why.\Pros: Requires no changes in existing implementations.\Cons: Inconsistent naming rules may confuse some users. Note, however, that this is the strategy that we took for MPI function names - when, in MPI-2, we came up with a naming strategy, we did not deprecate and remove the old functions (unless there was another reason to do so).
  4. MPI Implementation keys have a specified prefix as in 2, but existing keys are unchanged.\Pros: Requires no changes in existing implementations, leaves the name space "clean" for use of natural names like "file" for the standard.\Cons: Same as 3, but for the implementations.

My preference is for 3 or 4. I would vote against 1 or 2 because of the mess casued by having duplicate key names for the same property.

Finally, on the prefix. There is no confusion between strings and MPI objects. Thus a prefix of "mpi_" is perfectly adequate (for cases 1 and 3). "mpiinfo" as a prefix is too ugly and is unnecessary; there is no possibility of confusing "mpi_file" the string and mpi_file the object name.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-09-07 17:49:18 -0500

Thanks for the detailed comments, Bill.

I personally prefer options 1 and 3 (in that order). In fact, these were the two options we proposed at the start of this ticket. Two meetings ago, we took a straw-vote and option 1 was chosen (unanimously, in fact). If people are having second thoughts about it, we should do a straw-vote again in the December meeting (when there are enough organizations).

mpiforumbot commented 8 years ago

Originally by gropp on 2014-09-07 18:06:08 -0500

I believe the vote preferring 1 was taken before the problems of the duplicate key/values was discussed (without those problems, 1 is a clear choice and, as I noted, the one I would prefer if we were starting from scratch). As a result, I do think a new set of votes needs to be taken.

mpiforumbot commented 8 years ago

Originally by jsquyres on 2014-12-04 11:04:42 -0600

Bill -- I don't think that's right. I'm quite sure we talked about "file" and "mpi_file" both being set. See page 597:

An implementation must either provide both or neither of the old (deprecated) and new names of each of the predefined keys. For example, if a deprecated “keyname” is provided, then “mpikeyname” must also be provided and vice versa. If provided, the deprecated name is treated as an alias to the equivalent “mpi” key. That is, the value set by a key should be retrievable using the alias key and vice versa. Similarly, if a key and its alias are both set, the later set value overrides the former value for both the key and its alias.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-12-04 15:26:25 -0600

FYI, this ticket is scheduled for reading at this meeting. So there'll be a new set of votes. New straw votes at this meeting and formal votes at the later meetings. But Jeff is right -- both keys will be set. If people have objections, this meeting will be the time to speak up.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-12-10 18:27:02 -0600

Based on the F2F discussion at the Dec. 2014 meeting, the following ticket-0 changes were made:

diff --git a/trunk/working-groups/mpi40/ticket-411/chap-appLang/appLang-Const.tex b/trunk/working-groups/mpi40/ticket-411/chap-appLang/
index f17bbaa7..376de463 100644
--- a/trunk/working-groups/mpi40/ticket-411/chap-appLang/appLang-Const.tex
+++ b/trunk/working-groups/mpi40/ticket-411/chap-appLang/appLang-Const.tex
@@ -1457,6 +1457,13 @@ The following info keys are reserved. They are strings. \\[3pt]
 \infoskip{mpi\_striping\_factor} \\
 \infoskip{mpi\_striping\_unit} \\
 \infoskip{mpi\_wdir} \\
+\infoskip{mpi\_accumulate\_ops} \\
+\infoskip{mpi\_accumulate\_ordering} \\
+\infoskip{mpi\_argv} \\
+\infoskip{mpi\_command} \\
+\infoskip{mpi\_same\_size} \\
+\infoskip{mpi\_thread\_level} \\
+\infoskip{mpi\_maxprocs} \\
 \MPIupdateEnd{3.1}

 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
diff --git a/trunk/working-groups/mpi40/ticket-411/chap-deprecated/deprecated.tex b/trunk/working-groups/mpi40/ticket-411/chap-deprecat
index 097ac3ca..763e4c67 100644
--- a/trunk/working-groups/mpi40/ticket-411/chap-deprecated/deprecated.tex
+++ b/trunk/working-groups/mpi40/ticket-411/chap-deprecated/deprecated.tex
@@ -242,7 +242,6 @@ by equivalent info keys with the ``mpi\_'' prefix.
 \item[\infokey{io\_node\_list}] --- Superseded by \infokey{mpi\_io\_node\_list}.
 \item[\infokey{ip\_address}] --- Superseded by \infokey{mpi\_ip\_address}.
 \item[\infokey{ip\_port}] --- Superseded by \infokey{mpi\_ip\_port}.
-\item[\infokey{keyname}] --- Superseded by \infokey{mpi\_keyname}.
 \item[\infokey{maxprocs}] --- Superseded by \infokey{mpi\_maxprocs}.
 \item{\infokey{nb\_proc}} --- Superseded by \infokey{mpi\_nb\_proc}.
 \item{\infokey{no\_locks}} --- Superseded by \infokey{mpi\_no\_locks}.
diff --git a/trunk/working-groups/mpi40/ticket-411/chap-misc/misc-2.tex b/trunk/working-groups/mpi40/ticket-411/chap-misc/misc-2.tex
index 356047d8..39455e97 100644
--- a/trunk/working-groups/mpi40/ticket-411/chap-misc/misc-2.tex
+++ b/trunk/working-groups/mpi40/ticket-411/chap-misc/misc-2.tex
@@ -33,7 +33,7 @@ functionality.  An implementation is not required to support these
 keys and may support any others not reserved by \MPI/.
 \MPIupdate{3.1}{411}{However, if keys not specified by the \MPI/
   standard are used by the implementation, they must not start with
-  ``mpi\_'' (case sensitive).}
+  ``mpi\_'' (case insensitive).}

 An implementation must support info objects as caches for arbitrary (\mpiarg{key},\mpiarg{value}) pairs, regardless of whether it reco
 takes hints in the form of an \type{MPI\_Info} must be prepared to ignore any key it

New pdf is attached.

mpiforumbot commented 8 years ago

Originally by balaji on 2014-12-10 18:27:25 -0600

Attachment added: mpi-report.pdf (4236.1 KiB)

mpiforumbot commented 8 years ago

Originally by RolfRabenseifner on 2015-09-24 09:11:06 -0500

Replying to gropp: ...

  1. MPI Standard keys have a specified prefix (e.g., "mpi_"), and all existing keys in the standard are changed to follow this rule (e.g., "file" becomes "mpi_file").\Pros: What most people would have done in the beginning.\Cons: (Nearly?)everyone agrees that for backward compatibility, the old names must continue to be accepted, at least for a while (deprecated before being removed). This leads to many difficulties, as described above. E.g., how many keys are there? If the value of "file" is changed, does that change "mpi_file"? What if "file" is given one value and "mpi_file" another? And so on.

Copying has an additional problem according to MPI-3.1 p501:44 - p502:5. Will MPI_FILE_GET_INFO return

I fully agree.