search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 44 forks source link

Error in using spack-stack on Orion #807

Closed uturuncoglu closed 11 months ago

uturuncoglu commented 1 year ago

Describe the bug I am working on UFS Coastal App project and we are trying to create set of instructions for the users to install dependencies to their platform. For this purpose, I decide to start with the known platform (Orion) to have custom installation and run ufs-wather-model with it. But, I am getting following error related with the missing module when I try to install uff-weather model against my spack-stack installation. 

Lmod has detected the following error:  The following module(s) are unknown: "miniconda/3.9.7"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "miniconda/3.9.7"

Also make sure that all modulefiles written in TCL start with the string #%Module

Executing this command requires loading "miniconda/3.9.7" which failed while processing the following module(s):

    Module fullname     Module Filename
    ---------------     ---------------
    stack-python/3.9.7  /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/modulefiles/intel/2022.0.2/stack-python/3.9.7.lua
    ufs_orion.intel     /work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/modulefiles/ufs_orion.intel.lua
While processing the following module(s):
    Module fullname     Module Filename
    ---------------     ---------------
    stack-python/3.9.7  /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/modulefiles/intel/2022.0.2/stack-python/3.9.7.lua
    ufs_orion.intel     /work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/modulefiles/ufs_orion.intel.lua

To Reproduce Install space-stack:

- git clone --recursive https://github.com/JCSDA/spack-stack.git

- cd spack-stack

- git checkout 1.4.1

- git submodule update --init --recursive

- module load python

- source setup.sh (spack stack create env -h shows the list of available environments)

- create environment
spack stack create env --site orion --template unified-dev --name unified-dev.orion

- activate the newly created environment
cd envs/unified-dev.orion/
spack env activate .

- adjust environment if it is required (optional)
vi envs/unified-dev.orion/spack.yaml
vi envs/unified-dev.orion/common/*.yaml
vi envs/unified-dev.orion/site/*.yaml

- edit spack.yaml to keep only intel compiler and also ufs-weather-model-env@unified-dev
there is no need to install other environment for jedi etc.

- process/concretize the specs
spack concretize (it can be forced by spack concretize -f)

- install packages
spack install --source

- create module files
spack module lmod refresh

- create meta-modules for compiler, mpi, python
spack stack setup-meta-modules

Then modify modulefiles/ufs_orion.intel.lua file to point custom installation.

Expected behavior Expecting to build any configuration under ufs-weather-model.

System: What system(s) are you running the code on? Orion

Additional context We are plaining to integrate spack-stack with the ufs-coastal-app to provide more seamlessly way for the external users and contributors.

climbfuji commented 1 year ago

@ulmononian and @natalie-perlin Can you help with this support request on Orion?

ulmononian commented 1 year ago

@uturuncoglu can you share your modulefile? either paste here or the path on orion is ok.

my hunch is maybe you need to add this line to the modulefile:

prepend_path("MODULEPATH", "/work/noaa/da/role-da/spack-stack/modulefiles")

uturuncoglu commented 1 year ago

@ulmononian Thanks for your help. The top level module file is something like following,

(base) tufuk@Orion-login-2/work/noaa/nems/tufuk/COASTAL/ufs-coastal (feature/coastal_app) $ cat ../deps/ufs-coastal/modulefiles/ufs_orion.intel.lua
help([[
loads UFS Model prerequisites for Orion/Intel
]])

prepend_path("MODULEPATH", "/work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/modulefiles/Core")

stack_intel_ver=os.getenv("stack_intel_ver") or "2022.0.2"
load(pathJoin("stack-intel", stack_intel_ver))

stack_impi_ver=os.getenv("stack_impi_ver") or "2021.5.1"
load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))

stack_python_ver=os.getenv("stack_python_ver") or "3.9.7"
load(pathJoin("stack-python", stack_python_ver))

cmake_ver=os.getenv("cmake_ver") or "3.23.1"
load(pathJoin("cmake", cmake_ver))

load("ufs_common")

setenv("CC", "mpiicc")
setenv("CXX", "mpiicpc")
setenv("FC", "mpiifort")
setenv("CMAKE_Platform", "orion.intel")

whatis("Description: UFS build environment")
ulmononian commented 1 year ago

thank you @uturuncoglu. please add

prepend_path("MODULEPATH", "/work/noaa/da/role-da/spack-stack/modulefiles")

after your other prepend path line and let me know.

uturuncoglu commented 1 year ago

@ulmononian Thanks. It fixed the issue related to the module.

Here are my questions,

cd /work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/tests/build_fv3_coastal/CDEPS-interface && /apps/intel-2022.1.2/intel-2022.1.2/mpi/2021.5.1/bin/mpiifort  -I/work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/CDEPS-interface/ufs/cdeps_share -I/work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/tests/build_fv3_coastal/CDEPS-interface/mod -I/work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/esmf-8.4.2-s2eqx3y/include -I/work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-c-4.9.2-5ymrd45/include -I/work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/include -I/work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/parallelio-2.5.9-bn6l5jh/include -g -traceback -O -assume realloc_lhs -O2 -fp-model precise -module mod -c /work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/CDEPS-interface/CDEPS/share/shr_kind_mod.F90 -o CMakeFiles/share.dir/CDEPS/share/shr_kind_mod.F90.o
/work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/SCHISM-interface/SCHISM/src/Utility/UtilLib/extract_mod.f90(16): error #7013: This module file was not generated by any release of this compiler.   [NETCDF]
    use netcdf

This is one of the new component under ufs-coastal and was compiling with our any issue with the exiting spack-stack installation on Orion. It seems it could not find the netcdf library.

I also try to compile S2SWA application and it compiled without any issue. Just get following warnings at the end and it created executable,

ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
ld: warning: libnetcdf.so.15, needed by /work/noaa/nems/tufuk/COASTAL/deps/spack-stack/envs/unified-dev.orion/install/intel/2022.0.2/netcdf-fortran-4.6.0-5aiyu36/lib/libnetcdff.so, may conflict with libnetcdf.so.19
make[2]: Leaving directory `/work/noaa/nems/tufuk/COASTAL/deps/ufs-coastal/tests/build_fv3_coastal'

Then, I run cpld_control_p8 RT and it failed with following error,

 64: Warning! ***HDF5 library version mismatched error***
 64: The HDF5 header files used to compile this application do not match
 64: the version used by the HDF5 library to which this application is linked.
 64: Data corruption or segmentation faults may occur if the application continues.
 64: This can happen when an application was compiled by one version of HDF5 but
 64: linked with a different version of static or shared HDF5 library.
 64: You should recompile the application or check your shared library related
 64: settings such as 'LD_LIBRARY_PATH'.
 64: You can, at your own risk, disable this warning by setting the environment
 64: variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
 64: Setting it to 2 or higher will suppress the warning messages totally.
 64: Headers are 1.14.0, library is 1.10.5
 64:        SUMMARY OF THE HDF5 CONFIGURATION
 64:        =================================
 64:
 64: General Information:
 64: -------------------
natalie-perlin commented 1 year ago

@uturuncoglu - Additional modulefiles in /work/noaa/da/role-da/spack-stack/modulefiles/ are for spack-stack installed mysql, ecflow and miniconda (basic python-3.9.7 installation based on miniconda). These packages are not a part of spack-stack environment.

ulmononian commented 1 year ago

@uturuncoglu - Additional modulefiles in /work/noaa/da/role-da/spack-stack/modulefiles/ are for spack-stack installed mysql, ecflow and miniconda (basic python-3.9.7 installation based on miniconda). These packages are not a part of spack-stack environment.

thanks @natalie-perlin. also, for spack-stack 1.4.1, most supported platforms still utilized an external python to build spack-stack (which is why you need to add the /work/noaa/da/role-da/spack-stack/modulefiles path to load stack-python). we are looking to move away from external python dependencies entirely in the coming release(s).

we will have to look into the errors your reported, but none of us have utilized ufs coastal previously. if you built ufs-weather-model env from spack-stack/1.4.1, there were some changes needed for the ufs weather model to run. can you share the path to the ufs wm source where your modulefiles are located?

uturuncoglu commented 1 year ago

@natalie-perlin Thanks. How the process of spack-stack installation will be for totally custom platform? Is it possible to make them optional? Is there any way to install those too before installing spack stack. BTW, we could have complete control on custom platform.

climbfuji commented 1 year ago

We provide one set of vetted instructions, if users want to omit certain parts then they can certainly do that, but at their own risk (and we won't be able to provide support for that - limited resources).

uturuncoglu commented 1 year ago

@ulmononian The ufs-coastal is fork of ufs-wether-model has no any additional changes except having few new model components. At this point the development totally isolated and we did not tough any existing component build layer or component itself. So, In theory the cpld_control_p8 RT needs to be run without any issue but if you want me to try with fresh clone of ufs-weather-model I could do it.

@climbfuji That is totally fine. I know you all overbooked and lack of resource. The ufs-coastal has diverse community and partners also in Europe. So, having robust way to install dependencies would be critical for UFS Coastal. If there is something in your mind that could help to achieve that goal, I could also help in the development in spack-stack side. It would be slow in my side to implement something but I could learn quickly. Anyway, handling dependencies smoothly is one of the priority of the coastal project at this point. We might implement something in the app level but it would be hard for me to design it will lack of experience in spack-stack. So, the current work is just for learning the logic and then I might help to extend it for coastal application.

climbfuji commented 1 year ago

One thing that's going to happen in the next quarter is that we will try to reduce the number of external dependencies. Most importantly ecflow. The external Python installations are mostly gone already. The only thing left will be mysql, but since that's really only needed if you build ewok-env for JCSDA applications. We could get by with modifying the instructions and skipping the mysql parts when ewok-env is not used.

uturuncoglu commented 1 year ago

@climbfuji That is great news. It would be nice to have some packages. Anyway, I know this is an involving process and will be more stable in the near future. Thanks for all your help.

ulmononian commented 1 year ago

@ulmononian The ufs-coastal is fork of ufs-wether-model has no any additional changes except having few new model components. At this point the development totally isolated and we did not tough any existing component build layer or component itself. So, In theory the cpld_control_p8 RT needs to be run without any issue but if you want me to try with fresh clone of ufs-weather-model I could do it.

@climbfuji That is totally fine. I know you all overbooked and lack of resource. The ufs-coastal has diverse community and partners also in Europe. So, having robust way to install dependencies would be critical for UFS Coastal. If there is something in your mind that could help to achieve that goal, I could also help in the development in spack-stack side. It would be slow in my side to implement something but I could learn quickly. Anyway, handling dependencies smoothly is one of the priority of the coastal project at this point. We might implement something in the app level but it would be hard for me to design it will lack of experience in spack-stack. So, the current work is just for learning the logic and then I might help to extend it for coastal application.

can you please try to run cpld_control_p8 with ufs-wm develop, but with your stack/modulefile? as i mentioned, we needed to modify spack-stack/1.4.1 to include pio/2.5.10 and some flags for esmf/mapl, but (correct me if i am wrong @AlexanderRichert-NOAA), the pio version should not cause this issue and the esmf/mapl build flags were only to ensure bit-to-bit reproducibility, i.e., i think 1.4.1 should work out of the box.

uturuncoglu commented 1 year ago

@ulmononian Sure. Let me try. I'll update you about it.

uturuncoglu commented 1 year ago

@ulmononian Okay I run with ufs-weather-model develop. It compiled and run but failed in b2b checking (none of the files is identical),

baseline dir = /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/develop-20231002/cpld_control_p8_intel
working dir  = /work/noaa/stmp/tufuk/stmp/tufuk/FV3_RT/rt_392271/cpld_control_p8_intel
Checking test 001 cpld_control_p8_intel results ....
 Comparing sfcf021.tile1.nc ............ALT CHECK......NOT OK
 Comparing sfcf021.tile2.nc ............ALT CHECK......NOT OK
 Comparing sfcf021.tile3.nc ............ALT CHECK......NOT OK
 Comparing sfcf021.tile4.nc ............ALT CHECK......NOT OK

Any suggestion? I'll also sync ufs-coastal with uas-weather-model again and test to see what happens. I hope that one runs without HDF5 error. Not sure why baseline checking is failing since we are using same modules.

ulmononian commented 1 year ago

@uturuncoglu i expected the b2b checking to fail here. you'd have to re-build your stack w/ the esmf and mapl flags described in #712, which should allow b2b checks to pass.

i am unsure why you saw that hdf5 error in your fork of the ufs-wm, but hopefully this can resolve it for you.

uturuncoglu commented 1 year ago

@ulmononian Thanks for your great help. Yes, I'll try again with #712 and also sync the ufs-coastal again to see what happens. I'll update you later.

uturuncoglu commented 1 year ago

@ulmononian Okay. I have fresh clone in ufs-coastal and I could able to run the case without any issue with the new installation. It still fails in checking files but I'll try to fix it with https://github.com/JCSDA/spack-stack/issues/712 and test again. Thanks for your help.

ulmononian commented 1 year ago

@ulmononian Okay. I have fresh clone in ufs-coastal and I could able to run the case without any issue with the new installation. It still fails in checking files but I'll try to fix it with #712 and test again. Thanks for your help.

awesome! thanks for giving it a shot. let me know if you need help with the instructions in #712. btw: newer version so spack have these esmf/mapl flags by default (in case you move to a newer version).

uturuncoglu commented 1 year ago

@ulmononian @climbfuji I also tested this under Ubuntu/Docker and I could able to build the additional components provided by UFS Coastal without any issue. The only thing that I had issue is that I had to create module files for Python and OpenMPI since spack-stack did not create them for me after using spack module lmod refresh and spack stack setup-meta-modules commands (even if those modules are installed through the spack). Those modules files are used by the stack-openmpi and stack-python modules and build fails due to missing modules. Anyway, maybe this could be a bug in the spack-stack side and it would be nice to fix it.

climbfuji commented 1 year ago

Sounds a lot like version mismatches (actual compiler version versus the command you ran as part of the instructions)

uturuncoglu commented 1 year ago

@climbfuji Here is my entire workflow for your reference,

- prepare base environment
docker run -it ubuntu:latest
apt-get update
apt-get -y install unzip file gringo
apt-get -y install build-essential binutils-dev gfortran
apt-get -y install python3-dev
apt-get -y install cmake wget ca-certificates vim git
apt-get -y install lmod
lua_ver=$(which lua | xargs realpath -e | xargs basename)
apt-get -y install lib${lua_ver}-dev tcl-dev

- (optional) install newer version of module since default one is old
and issue with depends_on() calls in the *.lua files.

wget https://sourceforge.net/projects/lmod/files/Lmod-8.7.tar.bz2
tar xf Lmod-8.7.tar.bz2
cd Lmod-8.7
./configure --prefix=`pwd`
make install
. lmod/lmod/init/sh
module -v

Modules based on Lua: Version 8.7  2022-04-16 13:49 -05:00
    by Robert McLay mclay@tacc.utexas.edu

- clone spack-stack
cd /opt
git clone --recursive https://github.com/JCSDA/spack-stack.git
cd spack-stack
git checkout 1.4.1
git submodule update --init --recursive

- install dependencies
source setup.sh (note that spack stack create env -h shows the list of available environments)
spack stack create env --site linux.default --template ufs-weather-model --name ufs.local --prefix /opt/packages/ufs.local
spack env activate -p envs/ufs.local
spack concretize 2>&1 | tee log.concretize
spack install --source 2>&1 | tee log.install

- create module files
spack module lmod refresh
spack stack setup-meta-modules

- make available
module use /opt/packages/ufs.local/modulefiles/Core
module use /opt/packages/ufs.local/modulefiles

- create missing modules
- /opt/packages/ufs.local/modulefiles/gcc/11.4.0/python/3.10.8.lua
-- -*- lua -*-
-- Module file created by spack (https://github.com/spack/spack) on 2023-10-06 09:47:04.832109
--
-- python@3.10.8%gcc@11.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 build_system=generic build_type=Release arch=linux-ubuntu22.04-aarch64/rjflctc
--

whatis([[Name : python]])
whatis([[Version : 3.10.8]])
whatis([[Target : aarch64]])
whatis([[Short description : Library for Python]])
whatis([[Configure options : --prefix=/opt/packages/ufs.local/gcc/11.4.0/python-3.10.8-l5yn3uo --without-pydebug --enable-shared --without-ensurepip --with-openssl=/opt/packages/ufs.local/gcc/11.4.0/openssl-1.1.1t-vseknka --with-dbmliborder=gdbm --with-system-expat --with-system-ffi --enable-loadable-sqlite-extensions CFLAGS=-fPIC]])

help([[Name   : python]])
help([[Version: 3.10.8]])
help([[Target : aarch64]])
help()
help([[Library for Python]])

prepend_path("PATH", "/opt/packages/ufs.local/gcc/11.4.0/python-3.10.8-l5yn3uo/bin", ":")
prepend_path("MANPATH", "/opt/packages/ufs.local/gcc/11.4.0/python-3.10.8-l5yn3uo/share/man", ":")
prepend_path("LD_LIBRARY_PATH", "/opt/packages/ufs.local/gcc/11.4.0/python-3.10.8-l5yn3uo/lib", ":")

- /opt/packages/ufs.local/modulefiles/gcc/11.4.0/openmpi/4.1.5.lua
-- -*- lua -*-
-- Module file created by spack (https://github.com/spack/spack) on 2023-10-06 09:47:04.832109
--
-- openmpi@4.1.5%gcc@11.4.0~atomics~cuda~cxx~cxx_exceptions~gpfs+internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+romio+rsh~singularity+static+two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none arch=linux-ubuntu22.04-aarch64
--

whatis([[Name : openmpi]])
whatis([[Version : 4.1.5]])
whatis([[Target : aarch64]])
whatis([[Short description : Library for MPI]])
whatis([[Configure options : --enable-shared --disable-silent-rules --disable-builtin-atomics --enable-static --enable-mpi1-compatibility --without-psm --without-hcoll --without-mxm --without-psm2 --without-ofi --without-xpmem --without-cma --without-fca --without-verbs --without-knem --without-ucx --without-cray-xpmem --without-loadleveler --without-lsf --without-tm --without-alps --without-slurm --without-sge --disable-memchecker --with-libevent=/opt/packages/ufs.local/gcc/11.4.0/libevent-2.1.12-qua5vdp --with-pmix=/opt/packages/ufs.local/gcc/11.4.0/pmix-4.2.3-vzcficm --with-zlib=/opt/packages/ufs.local/gcc/11.4.0/zlib-1.2.13-jxno5sl --with-hwloc=/opt/packages/ufs.local/gcc/11.4.0/hwloc-2.9.0-s3zu7vo --disable-java --disable-mpi-java --with-gpfs=no --without-cuda --enable-wrapper-rpath --disable-wrapper-runpath --disable-mpi-cxx --disable-cxx-exceptions]])

help([[Name   : openmpi]])
help([[Version: 4.1.5]])
help([[Target : aarch64]])
help()
help([[Library for MPI]])

prepend_path("PATH", "/opt/packages/ufs.local/gcc/11.4.0/openmpi-4.1.5-fublk2b/bin", ":")
prepend_path("MANPATH", "/opt/packages/ufs.local/gcc/11.4.0/openmpi-4.1.5-fublk2b/share/man", ":")
prepend_path("LD_LIBRARY_PATH", "/opt/packages/ufs.local/gcc/11.4.0/openmpi-4.1.5-fublk2b/lib", ":")

- clone UFS coastal
mkdir /test
cd /test
git clone -b feature/coastal_app --recursive https://github.com/oceanmodeling/ufs-coastal.git

- edit module file and point path of dependency installations
cd ufs-coastal/modulefiles
create new module file called as ufs_local.gnu.lua with following content

---

help([[
loads UFS Model prerequisites for Linux/GNU
]])

prepend_path("MODULEPATH", "/opt/packages/ufs.local/modulefiles/Core")
prepend_path("MODULEPATH", "/opt/packages/ufs.local/modulefiles/gcc/11.4.0")

stack_gnu_ver=os.getenv("stack_gnu_ver") or "11.4.0"
load(pathJoin("stack-gcc", stack_gnu_ver))

stack_openmpi_ver=os.getenv("stack_openmpi_ver") or "4.1.5"
load(pathJoin("stack-openmpi", stack_openmpi_ver))

stack_python_ver=os.getenv("stack_python_ver") or "3.10.8"
load(pathJoin("stack-python", stack_python_ver))

cmake_ver=os.getenv("cmake_ver") or "3.23.1"
load(pathJoin("cmake", cmake_ver))

load("ufs_common")

setenv("CC", "mpicc")
setenv("CXX", "mpicxx")
setenv("F77", "mpif77")
setenv("F90", "mpif90")
setenv("FC", "mpif90")
setenv("CMAKE_Platform", "linux.gnu")

whatis("Description: UFS build environment")

- test build
cd tests
./compile.sh "local" "-DAPP=CSTLSW -DPDLIB=ON -DBUILD_TOOLS=ON -DNO_PARMETIS=OFF -DOLDIO=ON" coastal gnu NO NO
climbfuji commented 1 year ago 
spack stack create env --site linux.default --template ufs-weather-model --name ufs.local --prefix /opt/packages/ufs.local
spack env activate -p envs/ufs.local
spack concretize 2>&1 | tee log.concretize
spack install --source 2>&1 | tee log.install

If you look at our documentation, you are missing several steps in between those calls. The problem is that the landing page https://spack-stack.readthedocs.io/en/latest/CreatingEnvironments.html is totally misleading (@AlexanderRichert-NOAA FYI) because commands like spack stack setup-meta-modules won't work with them alone. The first sentence on the page reads

The following instructions are a simplified version of the instructions found in Sections 6.1 and 6.2, and should work for systems where spack-stack already has a site configuration, or where minimal configuration is necessary. which is wrong. They don't work on any of the systems, whether pre-configured or with minimal configuration. The links in this sentence, however, take you to the correct steps, for example for linux https://spack-stack.readthedocs.io/en/latest/NewSiteConfigs.html#newsiteconfigs-linux

On this page, especially step 7 in 6.2.3. is needed, but you have much better chances for success if you follow the steps in 6.2.1 or 6.2.2 (dependin on your OS) and then 6.2.3.

uturuncoglu commented 1 year ago

@climbfuji Thanks for your help. Sometimes documentation is confusing but based on my experience spack-stack is a giant leap in terms of simplifying the process of installing dependencies. Thanks to all. Let me try again to see what happens. Best,

uturuncoglu commented 12 months ago

@climbfuji Okay. It took time but I follow the instruction that you mention. I could able to load modules without any issue. Now, I am trying to use them under ufs-coastal. I'll update about it. Thanks again for your help.

uturuncoglu commented 12 months ago

@climbfuji I wonder if there is plan to sync spack submodule with the authoritative repository. I am having difficulty with one of the packages that is required for coastal app but issue fixed in the new version of spack. https://github.com/spack/spack/issues/40456

climbfuji commented 12 months ago

@uturuncoglu We can pull this version down, yes. It would be good if we (spack-stack developers) always pulled down any updates we make to spack develop. It's impossible to watch the spack repo, therefore I simply can't see what updates are going in.

uturuncoglu commented 12 months ago

@climbfuji Okay. Thanks. BTW, I could able to compile the components without any issue with new installation. Once I follow your instructions, I did not need to create python and openmpi modules by hand and everything was smooth. Thanks again for your help. If you want you could close this issue. Best,

climbfuji commented 12 months ago

@uturuncoglu See https://github.com/JCSDA/spack-stack/pull/834 and https://github.com/JCSDA/spack/pull/347 (currently draft PRs, once the CI tests all pass I'll open them up for review).