search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
25 stars 44 forks source link

Additional packages needed for NEPTUNE-JEDI #498

Closed areinecke closed 1 year ago

areinecke commented 1 year ago

Please describe the package or library you would like to add to spack-stack. We need to add additional packages to the NEPTUNE-JEDI environment for Narwhal, Nautilus, and Atlantis. The following pacakges are needed:

p4est v2.8 (https://p4est.github.io/release/p4est-2.8.tar.gz or https://github.com/cburstedde/p4est.git tag v2.8) Configuration options: --enable-mpi CFLAGS='-g -O1 -fPIC' CFLAGS='-g -O1 -fPIC' LIBS='-lz'

hdf5 Configuration options: --enable-parallel --enable-fortran --enable-fortran2003 FCFLAGS='-g -O3 -fPIC' CFLAGS='-g -O3 -fPIC'

nceplibs (for ccpp, single and double precision libraries)

yaml (https://github.com/yaml/libyaml tag 0.2.5)

ESMF (latest version: 8.xx)

What applications will be using this package or library? NEPTUNE, FALCON

Is there already a package or library in spack-stack that provides this, or related, functionality? Yes, but this is to add new dependencies that were not previously managed by spack.

Additional context Add any other context or screenshots about the package or library here.

Will This Package be Needed in a NOAA Operational Application? No

WCOSS System Software Request Checklist N/A

climbfuji commented 1 year ago

Thanks @areinecke ! We'll work on this after the upcoming release, which we expect to go out the door in the next 1-2 weeks.

climbfuji commented 1 year ago

@areinecke We had to get the spack-stack release 1.3.0 / 1.3.1 out the door, which took a bit longer than expected. I'll work on the additional libraries for NEPTUNE next week after EGU. Sorry for the delay!

climbfuji commented 1 year ago

@areinecke Can you post the updated/corrected libyaml env variables in this issue, please?

Also, how critical are the configuration options for the packages? The standard settings for p4est 2.8 in spack are equivalent to:

--enable-mpi --disable-openmp CFLAGS='-g -O2' LIBS='-lz'

(and by default, p4est is built as a shared library, therefore no -fPIC)

for hdf5 1.12.2:

--enable-parallel --enable-fortran FCFLAGS='-g -O2' CFLAGS='-g -O2'

(again, default build is shared; --enable-fortran2003 is not listed, therefore assuming it uses whatever the default is)

for libyaml, tag 2.5 is available.

esmf/mapl will be either be a combination of 8.3.0b09+2.22.0 or 8.4.2+2.35.2

and the various nceplibs of course, as needed by CCPP

climbfuji commented 1 year ago

@areinecke I found the libyaml settings (the corrected version you sent me):

      libyaml:
        environment:
          set:
            'YAML_ DIR': '{prefix}'
            'YAML_ LIB': '{prefix}/lib'
            'YAML_ INC': '{prefix}/include'
areinecke commented 1 year ago

@areinecke I found the libyaml settings (the corrected version you sent me):

      libyaml:
        environment:
          set:
            'YAML_ DIR': '{prefix}'
            'YAML_ LIB': '{prefix}/lib'
            'YAML_ INC': '{prefix}/include'

Yes, except no space between YAML and {DIR,LIB,INC}.

areinecke commented 1 year ago

@climbfuji,

On Atlantis we did the following: p4est@2.8+mpi~openmp libyaml@0.2.5 esmf@8.4.1+netcdf hdf5@1.12.1

Does that help?

climbfuji commented 1 year ago

@climbfuji,

On Atlantis we did the following: p4est@2.8+mpi~openmp libyaml@0.2.5 esmf@8.4.1+netcdf hdf5@1.12.1

Does that help?

Yes, thanks. I am running this on my Mac right now, if it works I'll try on Narwhal.

climbfuji commented 1 year ago

It built on macOS. I'll try on narwhal now.

climbfuji commented 1 year ago

@areinecke Can you see if this works for you, please?

module unload PrgEnv-cray
module load PrgEnv-intel/8.3.2
module unload intel
module load intel-classic/2021.4.0
module unload cray-mpich
module load cray-mpich/8.1.14
module unload cray-python
module load cray-python/3.9.7.1
module unload cray-libsci
module load cray-libsci/22.08.1.1

module use /p/app/projects/NEPTUNE/spack-stack/modulefiles
module load ecflow/5.8.4
module load mysql/8.0.31

module use /p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core
module load stack-intel/2021.4.0
module load stack-cray-mpich/8.1.14
module load stack-python/3.9.7
module load jedi-neptune-env/1.0.0

Which gives:

module li

Currently Loaded Modulefiles:
  1) craype/2.7.17                19) stack-cray-mpich/8.1.14      37) boost/1.78.0                 55) tar/1.30                     73) py-pycparser/2.21            91) py-pycodestyle/2.8.0
  2) craype-x86-rome              20) stack-python/3.9.7           38) bufr/11.7.1                  56) gettext/0.21.1               74) py-cffi/1.15.1               92) py-pyhdf/0.10.4
  3) libfabric/1.11.0.4.125       21) curl/7.66.0                  39) git-lfs/2.10.0               57) krb5/1.20.1                  75) py-findlibs/0.0.2            93) libyaml/0.2.5
  4) craype-network-ofi           22) pkg-config/0.29.2            40) crtm/v2.4.1-jedi             58) libtirpc/1.2.6               76) py-setuptools/59.4.0         94) py-pyyaml/6.0
  5) cray-dsmml/0.2.2             23) zlib/1.2.13                  41) ecbuild/3.7.2                59) hdf/4.2.15                   77) py-numpy/1.22.3              95) py-gast/0.5.3
  6) perftools-base/22.09.0       24) hdf5/1.12.2                  42) openjpeg/2.3.1               60) jedi-cmake/1.4.0             78) py-eccodes/1.4.2             96) py-beniget/0.4.1
  7) cray-pals/1.2.4              25) zstd/1.5.2                   43) eccodes/2.27.0               61) libpng/1.6.37                79) py-f90nml/1.4.3              97) py-ply/3.11
  8) bct-env/0.1                  26) netcdf-c/4.9.2               44) eigen/3.4.0                  62) libxt/1.1.5                  80) py-h5py/3.6.0                98) py-pythran/0.12.0
  9) mpscp/1.3a                   27) netcdf-fortran/4.6.0         45) openblas/0.3.19              63) libxmu/1.1.2                 81) py-cftime/1.0.3.4            99) py-scipy/1.9.3
 10) PrgEnv-intel/8.3.2           28) parallel-netcdf/1.12.2       46) eckit/1.23.0                 64) libxpm/3.5.12                82) py-netcdf4/1.5.3            100) py-xarray/2022.3.0
 11) intel-classic/2021.4.0       29) parallelio/2.5.9             47) fftw/3.3.10                  65) libxaw/1.0.13                83) py-bottleneck/1.3.5         101) sp/2.3.3
 12) cray-mpich/8.1.14            30) esmf/8.4.2                   48) fckit/0.10.1                 66) udunits/2.2.28               84) py-packaging/23.0           102) jedi-base-env/1.0.0
 13) cray-python/3.9.7.1          31) cmake/3.23.1                 49) fiat/1.1.0                   67) ncview/2.1.8                 85) py-numexpr/2.8.3            103) p4est/2.8
 14) cray-libsci/22.08.1.1        32) git/2.26.2                   50) ectrans/1.2.0                68) netcdf-cxx4/4.3.1            86) py-six/1.16.0               104) bacio/2.4.1
 15) qt/5.15.2                    33) nccmp/1.9.0.1                51) ecmwf-atlas/0.33.0           69) json/3.10.5                  87) py-python-dateutil/2.8.2    105) w3emc/2.9.2
 16) ecflow/5.8.4                 34) py-pip/21.2.3                52) gsibec/1.1.2                 70) json-schema-validator/2.1.0  88) py-pytz/2022.2.1            106) jedi-neptune-env/1.0.0
 17) mysql/8.0.31                 35) wget/1.20.3                  53) gsl-lite/0.37.0              71) odc/1.4.6                    89) py-pandas/1.4.0
 18) stack-intel/2021.4.0         36) base-env/1.0.0               54) libjpeg/2.1.0                72) py-attrs/22.2.0              90) py-pybind11/2.8.1

export | grep YAML

declare -x YAML_DIR="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27"
declare -x YAML_INC="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27/include"
declare -x YAML_LIB="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27/lib"
areinecke commented 1 year ago

I am getting errors with module use /p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core

reinecke@narwhal13:~> module unload PrgEnv-cray
reinecke@narwhal13:~> module load PrgEnv-intel/8.3.2
reinecke@narwhal13:~> module unload intel
reinecke@narwhal13:~> module load intel-classic/2021.4.0
reinecke@narwhal13:~> module unload cray-mpich
reinecke@narwhal13:~> module load cray-mpich/8.1.14
reinecke@narwhal13:~> module unload cray-python
reinecke@narwhal13:~> module load cray-python/3.9.7.1
reinecke@narwhal13:~> module unload cray-libsci
reinecke@narwhal13:~> module load cray-libsci/22.08.1.1
reinecke@narwhal13:~>
reinecke@narwhal13:~> module use /p/app/projects/NEPTUNE/spack-stack/modulefiles
reinecke@narwhal13:~> module load ecflow/5.8.4
reinecke@narwhal13:~> module load mysql/8.0.31
reinecke@narwhal13:~>
reinecke@narwhal13:~> module use /p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core
ModuleCmd_Use.c(240):ERROR:64: Directory '/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core' not found
reinecke@narwhal13:~> module load stack-intel/2021.4.0
ModuleCmd_Load.c(244):ERROR:105: Unable to locate a modulefile for 'stack-intel/2021.4.0'
reinecke@narwhal13:~> module load stack-cray-mpich/8.1.14
ModuleCmd_Load.c(244):ERROR:105: Unable to locate a modulefile for 'stack-cray-mpich/8.1.14'
reinecke@narwhal13:~> module load stack-python/3.9.7
ModuleCmd_Load.c(244):ERROR:105: Unable to locate a modulefile for 'stack-python/3.9.7'
reinecke@narwhal13:~> module load jedi-neptune-env/1.0.0
ModuleCmd_Load.c(244):ERROR:105: Unable to locate a modulefile for 'jedi-neptune-env/1.0.0'
climbfuji commented 1 year ago

ugh the default umask on narwhal is ridiculous

climbfuji commented 1 year ago

Fixing this now, sorry for that.

areinecke commented 1 year ago

Now I get:

module load stack-intel/2021.4.0
utility.c(2263):ERROR:50: Cannot open file '/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core/stack-intel/2021.4.0' for 'reading'
locate_module.c(591):ERROR:106: Magic cookie '#%Module' missing in '/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/modulefiles/Core/stack-intel/2021.4.0'
ModuleCmd_Load.c(244):ERROR:105: Unable to locate a modulefile for 'stack-intel/2021.4.0'
climbfuji commented 1 year ago

Well, this takes a while ... it's still running

climbfuji commented 1 year ago

Do you want to try again?

areinecke commented 1 year ago

Everything loaded, but a few requested directories are missing:

In common/modules.yaml can you remove libyaml from the blacklist and add the following:

      p4est:
        environment:
          set:
            'P4EST_API_DIR': '{prefix}'
            'P4EST_API_LIB': '{prefix}/lib'
            'P4EST_API_INC': '{prefix}/include'
      hdf5:
        environment:
          set:
            'HDF5_DIR': '{prefix}'
climbfuji commented 1 year ago

@areinecke Sure, I will do.

Note: I had already removed the libyaml from the blacklist, otherwise you wouldn't get the YAML_ env variables:

/p/work1/heinzell/spack-stack-add-neptune-packages> cat envs/jedi-neptune-env/common/modules.yaml | grep libyaml
climbfuji commented 1 year ago

Please try again (will need to reload all the modules):


heinzell@narwhal03:~> export | grep YAML
declare -x YAML_DIR="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27"
declare -x YAML_INC="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27/include"
declare -x YAML_LIB="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27/lib"
heinzell@narwhal03:~> export | grep P4EST
declare -x P4EST_API_DIR="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/p4est-2.8-23jgx22"
declare -x P4EST_API_INC="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/p4est-2.8-23jgx22/include"
declare -x P4EST_API_LIB="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/p4est-2.8-23jgx22/lib"
heinzell@narwhal03:~> export | grep HDF5_DIR
declare -x HDF5_DIR="/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/hdf5-1.12.2-xhsgy3e"
areinecke commented 1 year ago

At the final link stage I'm getting this permission denied:

/p/work1/heinzell/spack-stack-add-neptune-packages/spack/lib/spack/env/intel/ifort -auto  -o /p/home/reinecke/neptune_atmos8/bin/neptune_fcst.exe  /p/home/reinecke/neptune_atmos8/objdir/neptune_fcst.o /p/home/reinecke/neptune_atmos8/lib/libneptune.a -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/p4est-2.8-23jgx22/lib -lp4est -lsc -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/hdf5-1.12.2-xhsgy3e/lib -lhdf5_fortran -lhdf5 -ldl -lz -L/p/home/reinecke/neptune_atmos8/lib -lioapi -L/p/home/reinecke/neptune_atmos8/lib -ldsapi -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/libyaml-0.2.5-jbjzf27/lib -lyaml -lnuma -L/p/home/reinecke/neptune_atmos8/lib -lccpp_framework -lccpp_physics -L/lib -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/bacio-2.4.1-4gevzvt/lib -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/sp-2.3.3-6xotsxd/lib -lw3nco_d -lbacio_8 -lsp_d -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/netcdf-c-4.9.2-5xvc7ib/lib -lnetcdf -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/netcdf-fortran-4.6.0-rssh7ih/lib -lnetcdff -Wl,--no-as-needed  -qopenmp -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/esmf-8.4.2-cc2jvn6/lib -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/netcdf-c-4.9.2-5xvc7ib/lib -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/netcdf-fortran-4.6.0-rssh7ih/lib -L/p/work1/heinzell/spack-stack-add-neptune-packages/envs/jedi-neptune-env/install/intel/2021.4.0/parallelio-2.5.9-p6rs5n4/lib  -lesmf   -cxxlib -lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -r8  -g -traceback -cxxlib
gmake: execvp: /p/work1/heinzell/spack-stack-add-neptune-packages/spack/lib/spack/env/intel/ifort: Permission denied
gmake: *** [Makefile:206: /p/home/reinecke/neptune_atmos8/bin/neptune_fcst.exe] Error 127
climbfuji commented 1 year ago

Alright, that is still a remainder of that umask issue. Sorry. I can fix that when I am back at my desk in about 1.5hrs.

climbfuji commented 1 year ago

Please try again ...

areinecke commented 1 year ago

Now it's this error at linking:

[spack cc] ERROR: Spack compiler must be run from Spack! Input 'SPACK_ENV_PATH' is missing.
gmake: *** [Makefile:206: /p/home/reinecke/neptune_atmos8/bin/neptune_fcst.exe] Error 1
climbfuji commented 1 year ago

That is annoying. I wonder what else got messed up by the wrong umask. I gave group 4635GYFM full access to everything below /p/work1/heinzell/spack-stack-add-neptune-packages. Can you give it one last try, please? Not sure if it is better to log out and back in and rerun the module load commands.

If this doesn't work I'll install again with the correct umask in the project directory.

climbfuji commented 1 year ago

@areinecke I rebuilt everything in our project space overnight using umask 0022. The instructions stay the same, except one line:

module use /p/app/projects/NEPTUNE/spack-stack/spack-stack-add-neptune-pkgs/envs/jedi-neptune-env/install/modulefiles/Core
areinecke commented 1 year ago

I'm still getting the same error.

I'm confused as to why it's trying to use /p/app/projects/NEPTUNE/spack-stack/spack-stack-add-neptune-pkgs/spack/lib/spack/env/intel/ifort to link but everything else was built with ftn

climbfuji commented 1 year ago

Ok, thanks for trying. I am going to try jedi-bundle next.

climbfuji commented 1 year ago

Building on Neptune is a bit slow ... at 83%

climbfuji commented 1 year ago

My jedi-bundle build finished w/o errors, it used CC for the final linker step:

/p/work1/heinzell/jedi-bundle-20230511-jedi-neptune-env/build-release/log.make
areinecke commented 1 year ago

Okay, I see where were are running into a problem. For some reason we're reset the LD variable in our Makefile to ESMF_F90LINKER. If I don't do that it links fine. I'll have to understand why we're doing this.

climbfuji commented 1 year ago

Great, thanks for letting me know. It's good that it's not a spack-stack problem :-)