Running `singularity` workflow on other HPC platforms

[SorooshMani-NOAA]

So far the simple workflow implemented with Singularity containers and a Bash script is only tested on ParallelWorks platform. This needs to be tested on other HPC platforms as well. This Issue tracks problems related to that.

[SorooshMani-NOAA]

@WPringle I will soon share the Singularity images (.sif files) with you to test. Let's track all the issues (at high level) related to your tests here. Thanks again!

[WPringle]

@SorooshMani-NOAA You can put images on "Ensemble research collection" globus shared directory. You should have permissions to write.

[SorooshMani-NOAA]

@WPringle I have transferred the singularity images to your shared directory. In order to use them you need to download the https://github.com/oceanmodeling/ondemand-storm-workflow repo and switch to feature/singularity branch.

There is then a directory called singularity at the top level which includes image definitions as well as a scripts directory. In that scripts directory there is aworkflow.shand ainput.conffiles. In theory you should only modifyinput.confand runworkflow.sh` and the rest should be taken care of.

In input.conf file make sure to update:

• L_NWM_DATASET: Location of NWM hydrofabric (extracted) file (downloadable at https://www.nohrsc.noaa.gov/pub/staff/keicher/NWM_live/web/data_tools/NWM_channel_hydrofabric.tar.gz)
• L_TPXO_DATASET: Directory containing TPXO .nc files for pyschism to use
• L_DEM_HI: Directory containing high resolution DEMs (not used for subsetting)
• L_DEM_LO: Directory containing lower res DEMs, e.g. GEBCO
• L_MESH_HI: High resolution mesh file (e.g. .14) for subsetting
• L_MESH_LO: Low resolution mesh file (e.g. .14) for subsetting
• L_SHP_DIR: Directory containing precomputed domain shapes (not used for subsetting)
• L_IMG_DIR: Directory containing singularity images (.sif files I shared)
• L_SCRIPT_DIR: Path to the scripts directory where input.conf resides
• SINGULARITY_BINDFLAGS: Location bind flag for singularity, on an HPC you usually need to bind your scratch directory in order to have access to it from within the container, e.g. on ParallelWorks for me this would be "--bind /lustre"
• TMPDIR: Path to a location where temporary files can be stored by OCSMesh and PySCHISM while computing intermediate steps. Since home directory usually has limited space, I set this to some temp location on my scratch dir. For example on ParallelWorks I set it to /lustre/.tmp
• L_SOLVE_MODULES Name of the openmpi module on your HPC of choice, e.g. "openmpi/4.1.2"

Note that you need to make sure the HPC you're using has singularity command available. If it doesn't by default it might be under some module, that can for example be loaded by module load singularity or something like that.

Please let me know how it goes if you get the change to test this. Thanks!