search

ufs-community / ufs-srweather-app

UFS Short-Range Weather Application
Other
53 stars 113 forks source link

SRW modulefiles to use updated HPC-stack libraries and miniconda3 modules, on NOAA RDHPCs, in EPIC-managed common spaces #427

Closed natalie-perlin closed 1 year ago

natalie-perlin commented 1 year ago

Description

Modules of the srw-weather-app in ./modulefiles/ directory need to be updated to use the recent builds, in common EPIC-managed space, and to share locations with the UFS-Weather-Model. Miniconda3 modules with identical installations of the base conda and regional_workflow conda environments have been installed and tested on all the platforms. The environment regional_workflow contains all the packages for running the SRW, plotting figures using basic python examples, and met verification.

Additional updated hpc-stack libraries with the netcdf-c/4.9.0 have been build on Hera with several compiler/mpi options, for future testing. Other systems (Cheyenne, Gaea, Jet, Orion) are pending updates with netcdf-c/4.9.0, after successful testing on Hera is confirmed.

Locations of updated hpc-stack modules on the NOAA RDHPCs:

Cheyenne intel/2022.1: module load intel/2022.1 module load mpit/2.25 module use /glade/work/epicufsrt/contrib/hpc-stack/intel2022.1/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1 module load hpc-mpt/2.25

Cheyenne gnu/11.2: module gnu/11.2.0 module mpt/2.25 module use /glade/work/epicufsrt/contrib/hpc-stack/gnu11.2.0/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/11.2.0 module load hpc-mpt/2.25

Hera intel/2022.1.2 : module load intel/2022.1.2 module load impi/2022.1.2 use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/intel-2022.1.2/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1.2 module load hpc-impi/2022.1.2

Hera gnu/9.2 + mpich/3.3.2 : module load gnu/9.2 module use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-9.2/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/9.2 module load mpich/3.3.2 module load hpc-mpich/3.3.2

Hera gnu/10.2 + mpich/3.3.2 : module use /scratch1/NCEPDEV/nems/role.epic/gnu/modulefiles module load gnu/10.2.0 module use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-10.2/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/10.2 module load mpich/3.3.2 module load hpc-mpich/3.3.2

Hera gnu/10.2 + openmpi/4.1.2 : module use /scratch1/NCEPDEV/nems/role.epic/gnu/modulefiles module load gnu/10.2.0 module use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-10.2_openmpi/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/10.2 module load openmpi/4.1.2 module load hpc-openmpi/4.1.2

Hera gnu/9.2 + mpich/3.3.2 + netcdf 4.9: module load gnu/9.2 module use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-9.2_ncdf49/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/9.2 module load mpich/3.3.2 module load hpc-mpich/3.3.2

Hera gnu/10.2 + mpich/3.3.2 + netcdf/4.9: module use /scratch1/NCEPDEV/nems/role.epic/gnu/modulefiles module load gnu/10.2.0 module use /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-10.2_ncdf49/modulefiles/stack module load hpc/1.2.0 module load hpc-gnu/10.2 module load mpich/3.3.2 module load hpc-mpich/3.3.2

Gaea intel/2021.3.0: Lmod initialization on Gaea could to be done by sourcing the following script: source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh

module use /lustre/f2/dev/role.epic/contrib/hpc-stack/intel-2021.3.0/modulefiles/stack module load hpc/1.2.0 module load intel/2021.3.0 module load hpc-intel/2021.3.0 module load hpc-cray-mpich/7.7.11

Orion intel/2022.1.2: module load intel/2022.1.2 module load impi/2022.1.2 module use /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1.2 module load hpc-impi/2022.1.2

Jet intel/2022.1.2: module load intel/2022.1.2 module load impi/2022.1.2 module use /mnt/lfs4/HFIP/hfv3gfs/role.epic/hpc-stack/libs/intel-2022.1.2/modulefiles/stack module load hpc/1.2.0 module load hpc-intel/2022.1.2 module load hpc-impi/2022.1.2

Miniconda3/4.12.0 locations on the RDHPCs, with the regional_workflow environment created containing all the packages for running, plotting, and met verification.

Cheyenne: module use /glade/work/epicufsrt/contrib/miniconda3/modulefiles module load miniconda3/4.12.0 Hera: module use /scratch1/NCEPDEV/nems/role.epic/miniconda3/modulefiles module load miniconda3/4.12.0 Gaea: module use /lustre/f2/dev/role.epic/contrib/modulefiles module load miniconda3/4.12.0 Orion: module use /work/noaa/epic-ps/role-epic-ps/miniconda3/modulefiles module load miniconda3/4.12.0 Jet: module use /mnt/lfs4/HFIP/hfv3gfs/role.epic/miniconda3/modulefiles module load miniconda3/4.12.0

Solution

SRW modulefiles to be updated: ./modulefiles/build[.<compiler}>].lua ./modulefiles/wflow.lua - load miniconda3/4.12.0 and rocoto when needed ./modulefiles/tasks/machine/.local.lua - use regional_workflow environment installed under miniconda3/4.12.0

Requirements**

A shared location for software libraries for both ufs-srweather-app and ufs-weather-model to be used

Acceptance Criteria (Definition of Done)

Dependencies (optional)

ufs-srweather-app/PR-419

Similar/parallel Issue has been created in ufs-weather-model/issue-1465:

Alternative Solutions (optional)

MichaelLueken commented 1 year ago

The Cheyenne, Jet, and Orion modulefiles were updated to point to EPIC-managed locations at d036849. The Hera modulefiles were updated to point to EPIC-managed locations at d03e646. The Gaea modulefile was updated to point to EPIC-managed location at 550f400. Closing this issue now.