Informational: Spack config files for Cannon builds with Rocky 8.7

yantosca commented 1 year ago

Hi @pedmon and @maggiemcfee. I've been doing some experimenting with Spack builds on RockyLinux. I've created a Git repo with the modules.yaml, packages.yaml, and compilers.yaml files that seems to work well for both gcc 10.2.0 and gcc 12.2.0. Feel free to use this for documetnation purposes.

See: https://github.com/Harvard-ACMG/spack-env

Some notes:

In the packages.yaml I list the FASRC built modules (like CMake, hdf5, netcdf, netcdf-fortran, etc.) with external specs and also the buildable: False tag. This will prevent Spack from trying to rebuilld packages that are already present.
In modules.yaml, I've added settings so that module loads w/ Lmod will add the library path to LD_LIBRARY_PATH, and will also define some environment variables as Lmod does for FASRC-built modules

Let me know if you see anything that could be more streamlined.

yantosca commented 1 year ago

Still tweaking a few things and I will push updates later.

yantosca commented 1 year ago

Update 1 (02 May 2023)

I have not been able to get the Tuning and Analysis Utilities (TAU) version 2.3.2 to build. I've filed a new issue on the Spack Github.

Also tagging @jrwellshpc and @pkrastev

yantosca commented 1 year ago

Update 2 (03 May 2023)

I have verified that our mixed FASRC module / Spack Module environment works. See:

Spack configuration files at: https://github.com/Harvard-ACMG/spack-env/releases/tag/0.1.2
Module loading (aka "environment files) at https://github.com/Harvard-ACMG/cannon-env/tree/rocky-linux

I was able to build the Tuning and Analyis Utilities, but when I loaded it via a module, it invalidated the netcdf-fortran module (probably due to openmpi). I might build that separately.

yantosca commented 1 year ago

Update 3 (04 May 2023)

I have been trying to build a version of cdo that depends on netcdf-c and openmpi. However, it seems that the Lmod module aliasing will deactivate the netcdf-fortran module if I do module load cdo/2.1.0-gcc-12.2.0 (or 10.2.0).

Inactive Modules:
  1) netcdf-fortran/4.5.3-fasrc01

The following have been reloaded with a version change:
  1) hdf5/1.10.7-fasrc01 => hdf5/4.1.0-gcc-10.2.0
  2) netcdf-c/4.8.0-fasrc01 => netcdf-c/4.8.0-gcc-10.2.0
  3) openmpi/4.1.0-fasrc01 => openmpi/4.1.0-gcc-10.2.0
  4) zlib/1.2.11-fasrc01 => zlib/1.2.11-gcc-10.2.0

You can see this in the module avail output:

------- /n/sw/helmod-rocky8/modulefiles/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-fasrc01 --------
   fftw/3.3.10-fasrc01        netcdf-c/4.9.2-fasrc01       (L)
   hdf5/1.12.2-fasrc01        netcdf-cxx4/4.3.1-fasrc02
   hdf5/1.14.0-fasrc01 (L)    netcdf-fortran/4.6.0-fasrc02 (L)
   ior/3.3.0-fasrc01

------------------ /n/sw/helmod-rocky8/modulefiles/Comp/gcc/12.2.0-fasrc01 ------------------
   mpich/4.1-fasrc01    openmpi/4.1.4-fasrc01 (L)

---- /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/share/spack/lmod/linux-rocky8-x86_64/openmpi/4.1.4-6yi53gp/gcc/12.2.0 ----
   cdo/2.1.0-gcc-12.2.0   (D)    fftw/3.3.10-xac7a7n
   fftw/3.3.10-gcc-12.2.0        tau/2.32-lmfwamo    (D)

whereas netcdf-fortran depends on module openmpi/4.1.4-fasrc02 but cdo depends on openmpi/4.1.4-gcc-12.2.0. In reality both openmpi builds point to the same package (Spack assigns all external packages a Spack-like spec) but Lmod can't tell that apart. So when you load cdo, Lmod thinks you've loaded a different openmpi, which makes the netcdf-fortran an orphan.

Not sure if there is an easy way around this but @pkrastev is checking.

yantosca commented 1 year ago

One potential outcome is that we might have to have FASRC build as core modules any package that depends on MPI. It'd be nice if we can avoid this since that would be a lot of work

yantosca commented 1 year ago

There's also this issue on Spack: https://github.com/spack/spack/issues/3973

fasrc / User_Codes