Add scotch library to hpc-stack

[ulmononian]

A request to support the scotch library (on at least Hera and Orion) came from @aliabdolali via #501. This PR includes the pertinent additions/adaptations to enable installation of scotch as part of the stack, including:

• a new build_scotch.sh script in /libs
• the addition of "build_lib scotch" to the ufs 3rd party apps. section of build_stack.sh
• a scotch lua modulefile template

Successful installation as a part of a test hpc-stack was performed on Hera w/ Intel-2022.1.2 compiler and MPI (src: /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/src/scotchNsplenda , lib: /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch). @natalie-perlin will follow with Orion testing, as I do not have access.

I have not added a scotch section to any of the stack/*yaml files simply because I am not sure which would be most relevant. Open to suggestions, or we can just leave the inclusion of the scotch section to users who actually want/need the library in their stack.

Perhaps @aliabdolali may want to test loading the scotch-7.0.1 module to see if it works as expected in context (e.g., within WW3, etc.).

If the basic additions in this PR are deemed OK, we can go ahead and install scotch in the forthcoming, official, EPIC-maintained hpc-stack locations on Hera and Orion.

Note: I would like to recognize contributions from @natalie-perlin to this work.

[natalie-perlin]

A similar scotch.lua template modulefile is needed for the case of serial installation, in ./compiler/compilerName/compilerVersion/scotch/ . It is very similar to the one for the mpi, except for the part:

local hierA = hierarchyA(pkgNameVer,1) local compNameVer = hierA[1] local compNameVerD = compNameVer:gsub("/","-")

[ulmononian]

A similar scotch.lua template modulefile is needed for the case of serial installation, in ./compiler/compilerName/compilerVersion/scotch/ . It is very similar to the one for the mpi, except for the part:

local hierA = hierarchyA(pkgNameVer,1) local compNameVer = hierA[1] local compNameVerD = compNameVer:gsub("/","-")

done. please see: git commit -m "Add scotch modulefile ito hpc-stack/modulefiles/compiler/compilerName/compilerVersion for cases of serial installation.".

[edwardhartnett]

We should not be adding things to hpc-stack, this project is due to be retired very soon.

All new work should be happening is the spack-stack project.

[ulmononian]

We should not be adding things to hpc-stack, this project is due to be retired very soon.

All new work should be happening is the spack-stack project.

I understand this concern, @edwardhartnett . However, how are we to address requests for hpc-stack additions while developers are still utilizing this stack and spack-stack is not officially adopted or in use on the UFS-WM side? The transition process seems slightly nebulous.

[edwardhartnett]

Is this also being added to spack-stack?

[ulmononian]

Is this also being added to spack-stack?

It certainly can be (and the latest scotch is available through spack, as I tested an install of it: /scratch1/NCEPDEV/stmp2/Cameron.Book/spack_work/spack-stack/envs/scotch.intel), but a request to add it to a specific spack-stack environment/configuration has not been submitted.

[aliabdolali]

Hi @ulmononian

I tried WW3 with the scotch you compiled on Hera using the following modules:

module purge
  module load cmake/3.20.1
  module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
  module load hpc/1.2.0
  module load hpc-intel/2022.1.2
  module load hpc-impi/2022.1.2
  module load scotch
  module load netcdf/4.9.0
 module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0

We also need the following libraries, but I could not load them due to the conflict with the previously loaded libs. Could you guide how we should proceed?

module load bacio
module load w3emc
module load esmf

We also need to define SCOTCH_ROOT, similar to NETCDF_ROOT. Can we do it?

export SCOTCH_PATH=/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/

With the loaded libs and in absence of bacio, w3emc and esmf, I could run one test successfully that does not require those dependencies.

[climbfuji]

Is this also being added to spack-stack?

The issue is https://github.com/NOAA-EMC/spack-stack/issues/336 - nobody has worked on it yet.

[ulmononian]

Hi @ulmononian

I tried WW3 with the scotch you compiled on Hera using the following modules:

module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0

We also need the following libraries, but I could not load them due to the conflict with the previously loaded libs. Could you guide how we should proceed?

module load bacio
module load w3emc
module load esmf

We also need to define SCOTCH_ROOT, similar to NETCDF_ROOT. Can we do it?

export SCOTCH_PATH=/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/

With the loaded libs and in absence of bacio, w3emc and esmf, I could run one test successfully that does not require those dependencies.

@aliabdolali thanks for taking a look at this. let me get scotch installed in a stack with all of the libraries you mentioned. my intial test stack was simply to get scotch installed w/ its own dependencies, but i did not think about what scotch would need within ww3 for testing. my apologies!

i will also look into the SPACK_ROOT/SPACK_PATH issue that you mentioned. is this based on some issue you had when loading and running ww3 w/ the scotch module>

[edwardhartnett]

Please do not get something working in hpc-stack which is not also working in spack-stack.

Otherwise, when it is time to switch from hpc-stack to spack-stack, we will not be able to do so.

[ulmononian]

Please do not get something working in hpc-stack which is not also working in spack-stack.

Otherwise, when it is time to switch from hpc-stack to spack-stack, we will not be able to do so.

Fair enough, but how is this to be applied in practice when there is a request and need for software before the spack-stack is adopted by the UFS community? for example, what would the appropriate avenue for addressing #501, wherein ww3 development (and gfs downstream) requires scotch in an official stack?

[edwardhartnett]

Get it working in both hpc-stack and spack-stack. Getting it working only in hpc-stack is not good. We will not be able to retire hpc-stack if it contains software not available in spack-stack.

[ulmononian]

@aliabdolali i just added bacio, w3emc, and esmf (and their required dependencies) to /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack. i was able to load everything you mentioned w/:

module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0
module load bacio
module load w3emc
module load esmf

please let me know if this allows you to run scotch as required within ww3.

[ulmononian]

Get it working in both hpc-stack and spack-stack. Getting it working only in hpc-stack is not good. We will not be able to retire hpc-stack if it contains software not available in spack-stack.

that is understandable. i would be glad to also demonstrate that scotch works as part of of the spack-stack. as i said, i did install it already ( /scratch1/NCEPDEV/stmp2/Cameron.Book/spack_work/spack-stack/envs/scotch.intel), but if @edwardhartnett or @climbfuji have a specific installation config/testing in mind to validate scotch within spack-stack, i can try to handle it.

[aliabdolali]

@aliabdolali i just added bacio, w3emc, and esmf (and their required dependencies) to /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack. i was able to load everything you mentioned w/:

module purge
module load cmake/3.20.1
module use module use /scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/modulefiles/stack/
module load hpc/1.2.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2
module load scotch
module load netcdf/4.9.0
module load jasper/2.0.25
module load zlib/1.2.12
module load libpng/1.6.37
module load hdf5/1.12.2
module load g2lib/1.4.0
module load bacio
module load w3emc
module load esmf

please let me know if this allows you to run scotch as required within ww3.

Thanks @ulmononian I was able to test it successfully but would like WW3 code managers (@MatthewMasarik-NOAA @JessicaMeixner-NOAA) to confirm it would satisfy the rest of our tests. FYI: @AvichalMehra-NOAA

[ulmononian]

@aliabdolali as for the SCOTCH_ROOT question, i am not sure if i understand the issue. $scotch_ROOT is defined via

setenv("scotch_ROOT","/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1")

while its path is set via

prepend_path("PATH","/scratch1/NCEPDEV/stmp2/Cameron.Book/hpcs_work/libs/scotch/intel-2022.1.2/impi-2022.1.2/scotch/7.0.1/bin")

i've attached a screenshot to show what the paths/envars from the modulefile are read as for the current scotch location you are testing. please let me know if there is something done w/ the netcdf lua file that you need in the scotch lua file.

[ulmononian]

@aliabdolali: looking forward to hearing how the tests go for @MatthewMasarik-NOAA & @JessicaMeixner-NOAA.

[JessicaMeixner-NOAA]

@ulmononian we're waiting for some CMAKE issues using SCOTCH within WW3 to be resolved before we test this. Just wanted to keep you in that loop. We'll test this installation of SCOTCH as soon as those issues are resolved.

[ulmononian]

@JessicaMeixner-NOAA thanks for letting me know!

and speaking of cmake: @aliabdolali what was the reason you ended up including -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc in the cmake section of your build recipe? i have tested and can build on hera without hard-coding these compiler flags, but thought maybe you had a motivation for this.

[aliabdolali]

@JessicaMeixner-NOAA thanks for letting me know!

and speaking of cmake: @aliabdolali what was the reason you ended up including -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc in the cmake section of your build recipe? i have tested and can build on hera without hard-coding these compiler flags, but thought maybe you had a motivation for this.

There is no reason for it, I wanted to make sure icc and ifort are being used, I think we are good to go if you were able to compile it without them.

[aerorahul]

Should this be closed?

[natalie-perlin]

A scotch library 7.0.3 was rebuild and installed as a part of the hpc-stack on Hera and Orion, with the following options:

cmake VERBOSE=1 -DCMAKE_Fortran_COMPILER=mpiifort
-DCMAKE_C_COMPILER=mpiicc -DMAKE_CXX_COMPILER=mpiicpc -DCMAKE_INSTALL_PREFIX=/work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2/intel-2022.1.2/impi-2022.1.2/scotch/7.0.3 -DCMAKE_BUILD_TYPE=Release -DTHREADS=OFF -DMPI_THREAD_MULTIPLE=OFF ..

(https://github.com/NOAA-EMC/hpc-stack/issues/501#issuecomment-1542593588)

It cannot be easily added as a standard part of the hpc-stack, however, in particular for the current intel compilers, because it requires an updated gcc. The gcc as a part of intel compiler suite is outdated; gcc from GNU compiler suite could be used, but it requires modifying GNU modulefile to avoid conflicts with Intel compiler. Modified GNU compiler modulefiles were used on Orion and Hera. For the general user and not on NOAA RDHPCs, having a specific compiler as defined in config_custom.sh would be best.