Proposed revision of xSDK community package policy M4: portability to common architectures

[curfman]

xSDK community package policies: https://xsdk.info/policies

working version in Google docs: https://docs.google.com/document/d/1DCx2Duijb0COESCuxwEEK1j0BPe2cTIJ-AjtJxt3290/edit#heading=h.2hp5zbf0n3o3

In order to generalize policy language to encourage broad international collaboration on the xSDK, we propose to revise xSDK community policy M4 to remove the mention of specific computing facilities. This approach promotes general portability of xSDK packages to common architectures, while permitting subgroups of xSDK packages to pursue additional portability based on sponsor requirements (for example, DOE-sponsored packages targeting key machines at ALCF, NERSC, and OLCF). We thank international collaborators for pointing out the need for this policy revision.

Feedback on this proposed change is welcome by comments on this Github issue and/or in the working version of the xSDK policy document in Google docs (see hyperlink above).

Proposed new version M4. Each package team must do a “best effort” at portability to common architectures, including standard Linux distributions, GNU, Clang, and vendor compilers. Further portability requirements may be conditionally applied based on a package’s sponsor requirements. Support for Apple Mac OS and Microsoft Windows Visual Studio is recommended. Each package should document the versions of dependent packages with which it can work, preferably in a machine-readable form. The xSDK member packages will coordinate the needed versions of dependent packages for each xSDK release.

Current (old) version M4. Each package team must do a “best effort” at portability to key architectures, including standard Linux distributions, GNU, Clang, vendor compilers, and target machines at ALCF, NERSC, OLCF. Support for Apple Mac OS and Microsoft Windows Visual Studio is recommended. Each package should document the versions of dependent packages with which it can work, preferably in a machine-readable form. The xSDK member packages will coordinate the needed versions of dependent packages for each xSDK release.

[jedbrown]

What is the recommended machine-readable form for specifying dependencies? Also "X's dependent packages" would literally refer to packages that depend on X, not the packages that X depends on.

[tgamblin]

I think so far that has meant “make a spack package”, but there are things we could improve about dependency version specification in spack, and the fact that it’s embedded in Python doesn’t make it easy to archive outside of Spack.

Would a list of spack dependency specs in a YAML file do? The tricky parts are conditional dependencies, and dependencies that depend on the package version. For a single version of xSDK, though, the possible configurations are more limited.

[sslattery]

We have been spending some time in the DTK team parsing this and it seems as though M4 covers two separate topics such that something like this might make more sense:

Proposed new version M4. Each package team must do a “best effort” at portability to common architectures, including standard Linux distributions, GNU, Clang, and vendor compilers. Further portability requirements may be conditionally applied based on a package’s sponsor requirements. Support for Apple Mac OS and Microsoft Windows Visual Studio is recommended.

Proposed new M17. Each package should document the versions of dependent packages with which it can work, preferably in a machine-readable form. The xSDK member packages will coordinate the needed versions of dependent packages for each xSDK release.

At that point then M17 looks a lot like M16 although the proposed new M17 does not specifically mention package management.

Any comments or thoughts on why M4 should be either a single policy or split into two separate policies?

[mkstoyanov]

I agree with @sslattery. When policies cover multiple points, it is hard to assess compliance.

I also think that mixing "hard" and "soft" requirements can be misleading, e.g., "should document" but "preferably in machine-readable form", also "must do" vs. "is recommended". There should be once concise set of rules that everyone must follow, the "preferred" and "recommended" sections should go into the section of R policies.

[cswoodward]

I think splitting M4 into two as @sslattery suggests would be good. For M4 I further suggest removing "Further portability requirements may be conditionally applied based on a package’s sponsor requirements." as this comment is really saying that a package may add more portability as needed. Since these requirements are meant to be a minimal set, it seems strange to state sponsor-related needs of additional things here.

[curfman]

Based on these comments and discussion with the xSDK community, the working version of xSDK community package policies (in Google docs) has been revised as below. In a later phase of work (after the fall 2018 release), we will consider refinements of R6 (and possible transition to being a mandatory instead of recommended policy).

Feedback on this proposed change is welcome by further comments on this Github issue and/or in the working version of the xSDK policy document in Google docs.

Split policy M4 into 2 parts: M4 (portability to common platforms) and new policy R6 (package should document the versions of packages with which it can work on on which it depends).

Proposed new version M4. Each package team must do a “best effort” at portability to common platforms, including standard Linux distributions, and common compiler toolchains such as GNU, Clang, and vendor compilers. Further portability requirements for xSDK subsets may be conditionally applied based on sponsor requirements.* Support for Apple Mac OS and Microsoft Windows Visual Studio is recommended.

*footnote: For example, xSDK packages that receive funding from the mathematical libraries component of the U.S. Department of Energy Exascale Computing project must support portability to target machines at the computing facilities ALCF, NERSC, and OLCF.

Proposed new policy R6. Each package should document the versions of packages with which it can work it can work or upon which it depends, including software external to the xSDK, preferably in a machine-readable form. The developers of xSDK member packages will coordinate the needed versions of various packages for each xSDK release.