need library g2tmpl version 1.12.0 to use latest UPP with ufs-weather-model

SamuelTrahanNOAA commented 3 months ago

Description

The UPP, found in FV3/upp, now requires version 1.12.0 of the g2tmpl library. Spack Stack 1.6.0 only has version 1.10.2 in the set of modules accessible to ufs-weather-model. Until that is updated, we won't be able to update the inline post (FV3/upp).

Solution

Add that library version to systems that don't already have it. Most (all?) of them should already have it because UPP needs it. It's in a separate directory, though: upp-addon-env

Alternatives

None come to mind.

Related to

2321

SamuelTrahanNOAA commented 3 months ago

Pinging @WenMeng-NOAA @RatkoVasic-NOAA @jkbk2004

RatkoVasic-NOAA commented 3 months ago

@DomHeinzeller what is the approach when user need additional version of the certain library (like in this case) to already installed spack-stack?

SamuelTrahanNOAA commented 3 months ago

user need additional version

It is not just one user. Eventually, the UPP version in FV3 will have to update. Once it does, ufs-weather-model will need a newer g2tmpl.

climbfuji commented 3 months ago

@RatkoVasic-NOAA Since this library doesn't have any downstream dependencies, it should be easy.

Add new version to spack-stack develop
Backport to releases that need it
Install on all platforms and for all releases that need it

SamuelTrahanNOAA commented 3 months ago

I'm wary of opening a spack stack PR myself since that is typically maintained by EPIC.

Also, this wasn't an accident; somebody went to the trouble of adding a UPP set of libraries to Spack Stack. Surely they would have thought through the consequences of that. It is obvious that other Spack Stack environments would need that new library version to compile with UPP

If so, there is probably someone working on this problem already. Do you know who it is?

RatkoVasic-NOAA commented 3 months ago

I see on Hera new environment /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env from April 24th. Many people have access to that directory. @AlexanderRichert-NOAA did you start working on this?

AlexanderRichert-NOAA commented 3 months ago

Yes, this was something I implemented after discussion with @WenMeng-NOAA over email. See https://github.com/JCSDA/spack-stack/issues/1087

SamuelTrahanNOAA commented 3 months ago

You didn't implement it on Hera, though.

I tried upp-addon-env with ufs-weather-model and the latest UPP.

ufs_hera.intel.lua

```lua help([[ loads UFS Model prerequisites for Hera/Intel ]]) prepend_path("MODULEPATH", "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core") stack_intel_ver=os.getenv("stack_intel_ver") or "2021.5.0" load(pathJoin("stack-intel", stack_intel_ver)) stack_impi_ver=os.getenv("stack_impi_ver") or "2021.5.1" load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver)) cmake_ver=os.getenv("cmake_ver") or "3.23.1" load(pathJoin("cmake", cmake_ver)) load("ufs_common") nccmp_ver=os.getenv("nccmp_ver") or "1.9.0.1" load(pathJoin("nccmp", nccmp_ver)) setenv("CC", "mpiicc") setenv("CXX", "mpiicpc") setenv("FC", "mpiifort") setenv("CMAKE_Platform", "hera.intel") whatis("Description: UFS build environment") ```

ufs_common.lua

```lua whatis("Description: UFS build environment common libraries") help([[Load UFS Model common libraries]]) local ufs_modules = { {["jasper"] = "2.0.32"}, {["zlib"] = "1.2.13"}, {["libpng"] = "1.6.37"}, {["hdf5"] = "1.14.0"}, {["netcdf-c"] = "4.9.2"}, {["netcdf-fortran"] = "4.6.1"}, {["parallelio"] = "2.5.10"}, {["esmf"] = "8.6.0"}, {["fms"] = "2023.04"}, {["bacio"] = "2.4.1"}, {["crtm"] = "2.4.0"}, {["g2"] = "3.4.5"}, {["g2tmpl"] = "1.12.0"}, {["ip"] = "4.3.0"}, {["sp"] = "2.5.0"}, {["w3emc"] = "2.10.0"}, {["gftl-shared"] = "1.6.1"}, {["mapl"] = "2.40.3-esmf-8.6.0"}, {["scotch"] = "7.0.4"}, } for i = 1, #ufs_modules do for name, default_version in pairs(ufs_modules[i]) do local env_version_name = string.gsub(name, "-", "_") .. "_ver" load(pathJoin(name, os.getenv(env_version_name) or default_version)) end end ```

Some libraries are missing because you only installed the UPP requirements:

Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "esmf/8.6.0", "nccmp/1.9.0.1"
   Try: "module spider esmf/8.6.0 nccmp/1.9.0.1" to see how to load the module(s).

AlexanderRichert-NOAA commented 3 months ago

Sorry, I didn't read all of the previous exchange here; I meant only that the updated g2tmpl was there as part of those upp-addon-env's.

On Hera, I've added the rest of the UFS-related modules to upp-addon-env so you should be able to build UFS WM with g2tmpl@1.12.0.

SamuelTrahanNOAA commented 3 months ago

On Hera, I've added the rest of the UFS-related modules to upp-addon-env so you should be able to build UFS WM with g2tmpl@1.12.0.

The modules load. I'll try compiling and running momentarily.

SamuelTrahanNOAA commented 3 months ago

The compilation fails due to incompatibilities between UPP and the current version of FV3/io/post_fv3.F90. This is unrelated to the Spack Stack libraries. Someone needs to update the inline post to work with the latest UPP.

SamuelTrahanNOAA commented 3 months ago

I've documented the inline post code problem here:

https://github.com/NOAA-EMC/fv3atm/issues/846

And I am testing a fix for it.

SamuelTrahanNOAA commented 3 months ago

The gnu version of g2tmpl 1.12.0 is still missing.

It's not here at all:

/scratch2/NCEPDEV/stmp1/role.epic/spack-stack/spack-stack-1.6.0_gnu13/envs/ufs-wm-srw-rocky8/install/modulefiles/Core

And this directory is still using gnu 9.2.0 (!?) instead of 13.3.0:

/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core

SamuelTrahanNOAA commented 3 months ago

@AlexanderRichert-NOAA - You didn't update the gcc 13.3.0 spack stack to have g2tmpl 1.12.0. Both compilers must have g2tmpl 1.12.0 to run the ufs-weather-model.

AlexanderRichert-NOAA commented 3 months ago

Okay. I can add another modified environment with g2tmpl@1.12.0 based on gcc13.

AlexanderRichert-NOAA commented 3 months ago

Please try /scratch2/NCEPDEV/stmp1/role.epic/spack-stack/spack-stack-1.6.0_gnu13/envs/g2tmpl-addon-env/install/modulefiles/Core

SamuelTrahanNOAA commented 3 months ago

@AlexanderRichert-NOAA - Most of my group's resources are on GAEA C5, not Hera. Can you install updated modules on GAEA? Or if not, can you tell me how you're building Spack Stack 1.6.0 with the newer g2tmpl? I have my own setup to build Spack Stack on GAEA, and I can modify it if needed.

AlexanderRichert-NOAA commented 3 months ago

Just added /autofs/ncrc-svm1_proj/epic/spack-stack/spack-stack-1.6.0/envs/g2tmpl-addon-env/install/modulefiles/Core, please check it out

SamuelTrahanNOAA commented 3 months ago

Just added /autofs/ncrc-svm1_proj/epic/spack-stack/spack-stack-1.6.0/envs/g2tmpl-addon-env/install/modulefiles/Core, please check it out

I'm testing that in a model workflow and in the regression tests. When I have something to report, you'll know.

SamuelTrahanNOAA commented 3 months ago

@AlexanderRichert-NOAA - Can you add prod-util to your GAEA g2tmpl-addon-env? The workflow needs it for date manipulation.

AlexanderRichert-NOAA commented 3 months ago

Sure, please try again now

SamuelTrahanNOAA commented 3 months ago

@AlexanderRichert-NOAA - A later phase of the workflow wants grib-util. Can you add that on GAEA?

AlexanderRichert-NOAA commented 3 months ago

Done

AlexanderRichert-NOAA commented 2 months ago

My understanding is that @RatkoVasic-NOAA will deploy g2tmpl-addon-env on the remaining systems.

RatkoVasic-NOAA commented 2 months ago

@AlexanderRichert-NOAA I see you did it on Hera and Gaea. I'll do it on Jet, Hercules, Orion and Derecho.

WenMeng-NOAA commented 2 months ago

The g2tmpl/1.12.0 is available at upp-addon-env of the spack-stack-1.6.0 on Jet, Hercules and Orion as:

Jet: /mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/intel/2021.5.0
Orion: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/intel/2021.9.0
Hercules: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/intel/2021.9.0

RatkoVasic-NOAA commented 2 months ago

Thanks @WenMeng-NOAA , I'll check Derecho then. =edit= we are not installing addons on Derecho.

RatkoVasic-NOAA commented 1 month ago

Here are installations of new g2tmpl under upp addon on 6 machines:

Hera:
module use /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
module load stack-intel
module avail g2tmpl

Gaea:
module use /ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
module load stack-intel
module avail g2tmpl

Jet:
module use /mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core/
module load stack-intel
module avail g2tmpl

Orion:
module use /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
module load stack-intel
module avail g2tmpl

Hercules:
module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
module load stack-intel
module avail g2tmpl

Derecho:
module use /glade/work/epicufsrt/contrib/spack-stack/derecho/spack-stack-1.6.0/envs/upp-addon-env/install/modulefiles/Core
module load stack-intel
module avail g2tmpl

ufs-community / ufs-weather-model