Open SorooshMani-NOAA opened 1 year ago
@WPringle I will soon share the Singularity images (.sif
files) with you to test. Let's track all the issues (at high level) related to your tests here. Thanks again!
@SorooshMani-NOAA You can put images on "Ensemble research collection" globus shared directory. You should have permissions to write.
@WPringle I have transferred the singularity
images to your shared directory. In order to use them you need to download the https://github.com/oceanmodeling/ondemand-storm-workflow repo and switch to feature/singularity
branch.
There is then a directory called singularity
at the top level which includes image definitions as well as a scripts
directory. In that scripts directory there is a
workflow.shand a
input.conffiles. In theory you should only modify
input.confand run
workflow.sh` and the rest should be taken care of.
In input.conf
file make sure to update:
L_NWM_DATASET
: Location of NWM hydrofabric (extracted) file (downloadable at https://www.nohrsc.noaa.gov/pub/staff/keicher/NWM_live/web/data_tools/NWM_channel_hydrofabric.tar.gz)L_TPXO_DATASET
: Directory containing TPXO .nc
files for pyschism
to useL_DEM_HI
: Directory containing high resolution DEMs (not used for subsetting)L_DEM_LO
: Directory containing lower res DEMs, e.g. GEBCO L_MESH_HI
: High resolution mesh file (e.g. .14
) for subsettingL_MESH_LO
: Low resolution mesh file (e.g. .14
) for subsettingL_SHP_DIR
: Directory containing precomputed domain shapes (not used for subsetting)L_IMG_DIR
: Directory containing singularity images (.sif
files I shared)L_SCRIPT_DIR
: Path to the scripts directory where input.conf
residesSINGULARITY_BINDFLAGS
: Location bind flag for singularity, on an HPC you usually need to bind your scratch directory in order to have access to it from within the container, e.g. on ParallelWorks for me this would be "--bind /lustre"
TMPDIR
: Path to a location where temporary files can be stored by OCSMesh and PySCHISM while computing intermediate steps. Since home directory usually has limited space, I set this to some temp location on my scratch dir. For example on ParallelWorks I set it to /lustre/.tmp
L_SOLVE_MODULES
Name of the openmpi
module on your HPC of choice, e.g. "openmpi/4.1.2"Note that you need to make sure the HPC you're using has singularity
command available. If it doesn't by default it might be under some module, that can for example be loaded by module load singularity
or something like that.
Please let me know how it goes if you get the change to test this. Thanks!
So far the simple workflow implemented with Singularity containers and a Bash script is only tested on ParallelWorks platform. This needs to be tested on other HPC platforms as well. This Issue tracks problems related to that.