Prepare and pursue Chapel's application to E4S

[bonachea]

Now that Chapel finally has a high-quality Spack package, the next logical step is to pursue Chapel membership in the E4S Programming Model SDK.

E4S is a DOE ASCR product that provides pre-built container images with selected Spack packages for a variety of systems, and (perhaps more importantly) professionally managed deployments on DOE production HPC centers. Getting Chapel into E4S is a relatively low-effort approach to have it incorporated into the modules installed across a large set of important systems.

You can read about the E4S membership criteria here. Off-hand I suspect that Chapel already roughly meets many of the required criteria, but a few (notably including P2 and P5) will probably require some minor additional effort. It's worth noting that many of these criteria come down to a judgement call. As such, I suggest it's in Chapel's best interest to ensure we're prepared to make a strong and coordinated case for how Chapel fulfills each of the required criteria, before proceeding with an application for E4S membership.

I'd like to be involved in this process.

[arezaii]

I took a preliminary pass at comparing the requirements to the project's state. Here is what I think we're starting from, roughly:

• [x] P1 Spack-based Build and Installation - Our spack package builds and installs the compiler and runtime, with several variants for configuring the installation.
• [x] P2 Minimal Validation Testing - TBD - basically contributing a test for the Chapel spackage to the e4s testing repo. PR (needs review) https://github.com/E4S-Project/testsuite/pull/74
• [x] P3 Sustainability - The Chapel sources for the spackage are the same as Chapel's normal release and share a release cycle cadence of once every 3 months.
• [x] P4 Documentation - The Chapel project documentation is available at https://chapel-lang.org/docs/
• [x] P5 Product Metadata - TBD - need to provide some brief info to populate the fields on this table (https://e4s-project.github.io/DocPortal.html) PR accepted at https://github.com/E4S-Project/E4S-Documenter/pull/9
• [x] P6 Public Repository - Chapel sources are available from the public repository github.com/chapel-lang/chapel
• [ ] P7 Imported Software - TBD - we mostly accomplish this by providing spack variants for hwloc, llvm, jemalloc, etc
• [x] P8 Error Handling - Chapel provides helpful error messages
• [x] P9 Test Suite - Chapel provides support for make check through spack test run chapel (need more tests than that)

[bonachea]

PRs against E4S repositories associated with P2 and P5 above are now merged, and Chapel now appears in the online E4S product list as a "Member Product".

Our understanding is that Chapel builds should first appear in the E4S 24.11 release that will be packaged in November.

I concur with @arezaii that P7 is also mostly accomplished, as the Spack package variants currently allow swapping in alternative versions of the following bundled third-party dependencies:

• gasnet
• gmp
• hwloc
• jemalloc
• libfabric
• libunwind
• llvm

This list includes all the dependencies for which the Chapel build system currently documents support for =system settings.

Dependencies currently missing =system build system support and variants in the chapel spackage include:

• qthreads
• re2

Both of these are available in Spack, and could be added as variants with additional effort. However it appears bundled versions of both libraries include some Chapel-specific patches, so vanilla versions might not function correctly without additional Spack-driven patching.

[bradcray]

Of those two packages, the re2 patches are necessary for using re2 with Chapel's IO features, and we were unsuccessful in getting them accepted back upstream when we tried (many years ago now, admittedly).

qthreads feels like a package where we'd have a much better chance of using the official Spack formula—I believe that most qthreads patches have traditionally been bug fixes that we've applied before they were available in an official release, though I know we added one recently to work around a C/C++ issue (https://github.com/chapel-lang/chapel/pull/25507 which may be addressed in future qthreads releases, though I don't think we know yet). And we also recently updated our bundled version to match Qthreads' latest release, which I believe removed the need for any previous bug fix patches (https://github.com/chapel-lang/chapel/pull/25210). @jabraham17 would be the expert on the latter, and would also know whether we carried forward the patch from #25507.

[jabraham17]

Yes we did carry forward the patch from #25507, and currently that is the only patch applied to qthreads on main today.