Trouble running release v2.2.0 container on some tier 1 platforms

[EdwardSnyder-NOAA]

Expected behavior

The previous release (v2.1.0) container was able to run on all T1 platforms using the instructions listed in the documentation.

Current behavior

The current release container (ubuntu20.04-intel-srwapp-release-public-v2.2.0.img) doesn't run on Cheyenne/Derecho, Gaea, and Orion/Hercules using the instructions mentioned in the previous section.

Machines affected

• Cheyenne/Derecho is failing in the make_sfc_climo step with a forrtl error.

 - CALL FieldRegridStore.
 - CALL FieldRegridStore.
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
sfc_climo_gen      0000000001837F1B  Unknown               Unknown  Unknown

• Gaea is failing in the make_sfc_climo step with a permission error FATAL ERROR: ERROR IN NF90_CREATE: Permission denied, but after updating the srun command to srun --mpi=pmi2, the PET log files reveal that the process is hanging on FieldRegridStore. One way to remedy this is to reduce the number of cores or to run on a single node. Unfortunately, no combination worked for me. So I ran this step on one node with the time command, which worked. I then ran the make_ics and make_lbcs steps with the time command successfully on a single node. However, the run_fcst step fails with the time command.
• Orion/Hercules will fail in get_extrn_ics and get_extrn_lbcs steps because it cannot find the proper python. This can be mitigated if we add the python path in the build modulefile. After that, it will fail in the make_sfc_climo step but will pass if we run it with the mpiexec -np $nprocs command. I was able to get the remaining steps (make_ics, make_lbcs, run_fcst, and run_post) to pass by running them on a single node with the mpiexec executable path appended to the PATH var in each one of these steps exregional scripts. Making these changes will lead to the community case to pass on Orion.

I think all these issues are related and are all caused by conflicts between the native intel/mpi environment and the intel/mpi environment in the container. Also Orion has some sort of access or permissions issue because it has trouble seeing the platform variables like mpiexec and python.

Steps To Reproduce

1. Pick one of the platforms mentioned above.
2. Follow the instructions mentioned here.
3. Once the experiment is built, run tasks with rocotorun until you encounter an error.

[EdwardSnyder-NOAA]

The release container can run on Hercules and Orion with the changes listed below. These changes are needed to run the develop container on these platforms as well.

Hercules

• compiler and mpi used to run the experiment were: intel-oneapi-compilers/2022.2.1 and intel-oneapi-mpi/2021.7.1
• added the python path to the jobs scripts in the same fashion we do for the exregional scripts
• added nco/4.9.3 to build file for make_grid step
• switched RUNCMD* to mpiexec -n $nprocs
• switched run_post to 4 nodes with 12 processors and increased time to 45 mins which has an avg completion time of 30 mins
• increased make_sfc_climo time to 50 mins with avg completion of 42 mins

Orion

• compiler and mpi used to run the experiment were: intel/2022.1.2 and impi/2022.1.2
• added the python path to the jobs scripts in the same fashion we do for the exregional scripts
• added nco/4.9.3 to build file for make_grid step
• switched RUNCMD* to mpiexec -n $nprocs
• switched run_post config to 4 nodes with 12 processors and increased time to 45 mins which has an avg completion time of 17 mins

[EdwardSnyder-NOAA]

Gaea is currently experiencing Lustre security issues, so container use on Gaea was disabled (i.e. singularity commands don't work). The containers will be enabled when the issue is patched, or when the new F5 file system is available. In addition, container use on Gaea is in a beta state.