search

JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 45 forks source link

[INSTALL] parallelio 2.5.10 for ufs-weather-model-env #701

Closed AlexanderRichert-NOAA closed 1 year ago

AlexanderRichert-NOAA commented 1 year ago

Once we confirm that no other changes are needed, we need to do an interim installation of ufs-weather-model dependencies that uses parallelio@2.5.10 for https://github.com/ufs-community/ufs-weather-model/pull/1745 using spack-stack-1.4.1 as the basis (i.e., create another environment under the existing spack-stack-1.4.1 envs/ directory on each system).

Instructions:

spack stack create env --name ufs-pio-2.5.10 --template empty --site <site>
cd envs/ufs-pio-2.5.10
spack env activate .
spack add ufs-weather-model-env%intel ^parallelio@2.5.10 # add second entry for gcc as needed
echo -e "  upstreams:\n    unified-env-1.4.1:\n      install_tree: /path/to/install" >> spack.yaml
spack concretize | tee log.concretize
# Confirm that only parallelio, esmf, and mapl (plus base-env/ufs-weather-model-env metapackages) are getting updated; everything else in concretize output should have '[^]'
spack install | tee log.install
spack module lmod refresh --upstream-modules
spack stack setup-meta-modules

Systems:

AlexanderRichert-NOAA commented 1 year ago

This is approximately what the output should look like (note the [^] for upstream packages):

==> Concretized ufs-weather-model-env%intel ^parallelio@2.5.10
 -   see3rjc  ufs-weather-model-env@1.0.0%intel@19.1.3.304~debug+python build_system=bundle arch=linux-sles15-zen2
[^]  rlfvsu3      ^bacio@2.4.1%intel@19.1.3.304~ipo+pic build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  padsq4n          ^cmake@3.23.1%intel@19.1.3.304~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-sles15-zen2
[^]  r7drzq6              ^ncurses@6.4%intel@19.1.3.304~symlinks+termlib abi=none build_system=autotools arch=linux-sles15-zen2
[^]  hqthlth              ^openssl@1.1.1t%intel@19.1.3.304~docs+shared build_system=generic certs=mozilla arch=linux-sles15-zen2
[^]  hdfx2bu                  ^ca-certificates-mozilla@2023-01-10%intel@19.1.3.304 build_system=generic arch=linux-sles15-zen2
[^]  2lfnkwl          ^gmake@4.4.1%intel@19.1.3.304~guile build_system=autotools arch=linux-sles15-zen2
 -   mbchtbo      ^base-env@1.0.0%intel@19.1.3.304 build_system=bundle arch=linux-sles15-zen2
[^]  6keecbo          ^curl@8.0.1%intel@19.1.3.304~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen2
[^]  eyz7fdy              ^pkg-config@0.29.2%intel@19.1.3.304+internal_glib build_system=autotools arch=linux-sles15-zen2
[^]  zyh3dps          ^git@2.35.3%intel@19.1.3.304+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-sles15-zen2
[^]  2mozkbq          ^hdf5@1.14.0%intel@19.1.3.304~cxx+fortran+hl~ipo~java+mpi+shared+szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make patches=0b5dd6f arch=linux-sles15-zen2
[^]  othsdey              ^libaec@1.0.6%intel@19.1.3.304~ipo+shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  qmbjrle          ^nccmp@1.9.0.1%intel@19.1.3.304~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  kcnfazd          ^netcdf-c@4.9.2%intel@19.1.3.304~byterange+dap~fismahigh~fsync~hdf4~jna+mpi+nczarr+optimize~parallel-netcdf+pic+shared+zstd build_system=autotools arch=linux-sles15-zen2
[^]  d3jbtqt              ^libxml2@2.10.3%intel@19.1.3.304~pic~python+shared build_system=autotools arch=linux-sles15-zen2
[^]  q5chqll                  ^xz@5.4.1%intel@19.1.3.304~pic build_system=autotools libs=shared,static arch=linux-sles15-zen2
[^]  ixnrcrh              ^m4@1.4.19%intel@19.1.3.304+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-sles15-zen2
[^]  ndc3de6                  ^libsigsegv@2.14%intel@19.1.3.304 build_system=autotools arch=linux-sles15-zen2
[^]  ssitvbp              ^zstd@1.5.2%intel@19.1.3.304~ipo+programs build_system=cmake build_type=RelWithDebInfo compression=none generator=make libs=shared,static arch=linux-sles15-zen2
[^]  p5o4oan          ^netcdf-fortran@4.6.0%intel@19.1.3.304~doc+pic+shared build_system=autotools arch=linux-sles15-zen2
[^]  nwb7voi          ^parallel-netcdf@1.12.2%intel@19.1.3.304~burstbuffer+cxx+fortran+pic+shared build_system=autotools arch=linux-sles15-zen2
[^]  5pbpbxm              ^perl@5.26.1%intel@19.1.3.304~cpanm+open+shared+threads build_system=generic patches=0eac10e,8cf4302 arch=linux-sles15-zen2
 -   d6ynrke          ^parallelio@2.5.10%intel@19.1.3.304+fortran~ipo~logging+mpi+pnetcdf+shared~timing build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  owu3x4l          ^py-pip@23.0%intel@19.1.3.304 build_system=generic arch=linux-sles15-zen2
[^]  3c7zodg          ^python@3.8.6%intel@19.1.3.304+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-sles15-zen2
[^]  npeu4il          ^wget@1.21.2%intel@19.1.3.304~libpsl~pcre~python+zlib build_system=autotools ssl=openssl arch=linux-sles15-zen2
[^]  s5h77fy              ^gettext@0.19.7%intel@19.1.3.304+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools patches=9acdb4e arch=linux-sles15-zen2
[^]  5le2wth                  ^bzip2@1.0.8%intel@19.1.3.304~debug~pic+shared build_system=generic arch=linux-sles15-zen2
[^]  k4wj2dv                  ^tar@1.34%intel@19.1.3.304 build_system=autotools zip=pigz arch=linux-sles15-zen2
[^]  xiq35gw                      ^pigz@2.7%intel@19.1.3.304 build_system=makefile arch=linux-sles15-zen2
[^]  wxw7nrq              ^libiconv@1.17%intel@19.1.3.304 build_system=autotools libs=shared,static arch=linux-sles15-zen2
[^]  74d3okx          ^zlib@1.2.13%intel@19.1.3.304+optimize+pic+shared build_system=makefile arch=linux-sles15-zen2
[^]  rbb3ovu      ^crtm@2.4.0%intel@19.1.3.304+fix~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  5rbf7lv          ^crtm-fix@2.4.0_emc%intel@19.1.3.304+big_endian~little_endian+netcdf build_system=generic arch=linux-sles15-zen2
[^]  nqhvyes          ^git-lfs@2.11.0%intel@19.1.3.304 build_system=makefile arch=linux-sles15-zen2
 -   por67on      ^esmf@8.4.2%intel@19.1.3.304~debug~external-lapack+external-parallelio+mpi+netcdf~pnetcdf~shared~xerces build_system=makefile esmf_comm=mpich3 esmf_os=Linux esmf_pio=auto patches=f63d405 snapshot=b09 arch=linux-sles15-zen2
[^]  zhz3fn4          ^cray-mpich@8.1.9%intel@19.1.3.304~wrappers build_system=generic arch=linux-sles15-zen2
[^]  uiitoco      ^fms@2023.01%intel@19.1.3.304+gfs_phys~ipo+openmp+pic+quad_precision~yaml build_system=cmake build_type=Release constants=GFS generator=make precision=32,64 arch=linux-sles15-zen2
[^]  hdaoete      ^g2@3.4.5%intel@19.1.3.304~ipo+pic build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  5znjjrh          ^jasper@2.0.32%intel@19.1.3.304+jpeg~opengl+shared build_system=generic build_type=Release arch=linux-sles15-zen2
[^]  bell45a              ^libjpeg-turbo@2.1.0%intel@19.1.3.304~ipo~jpeg8+pic+shared+static build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  vtnwczf                  ^nasm@2.15.05%intel@19.1.3.304 build_system=autotools arch=linux-sles15-zen2
[^]  s6zgkie          ^libpng@1.6.37%intel@19.1.3.304~ipo~pic build_system=cmake build_type=RelWithDebInfo generator=make libs=shared,static arch=linux-sles15-zen2
[^]  jiuagkl      ^g2tmpl@1.10.2%intel@19.1.3.304~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  txgtete      ^ip@3.3.3%intel@19.1.3.304~ipo+openmp+pic build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
 -   gs7uozx      ^mapl@2.35.2%intel@19.1.3.304~debug+esma_gfe_namespace~extdata2g~fargparse~flap~ipo~pflogger+pnetcdf~shared build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  qfwcbi4          ^ecbuild@3.7.2%intel@19.1.3.304~fismahigh~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  57l3e6y          ^gftl-shared@1.5.0%intel@19.1.3.304~ipo build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[^]  yc64hit              ^gftl@1.8.3%intel@19.1.3.304~ipo build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[^]  gdxl4uw          ^yafyaml@0.5.1%intel@19.1.3.304~fismahigh~ipo build_system=cmake build_type=Release generator=make arch=linux-sles15-zen2
[^]  lygv3cz      ^scotch@7.0.3%intel@19.1.3.304+compression~esmumps~int64~ipo+metis+mpi~mpi_thread+noarch~shared~threads build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  j7z5z56          ^bison@3.8.2%intel@19.1.3.304 build_system=autotools arch=linux-sles15-zen2
[^]  rvbxqfe              ^diffutils@3.9%intel@19.1.3.304 build_system=autotools arch=linux-sles15-zen2
[^]  eakmqwr          ^flex@2.6.4%intel@19.1.3.304+lex~nls build_system=autotools patches=f8b85a0 arch=linux-sles15-zen2
[^]  gyh7eic      ^sp@2.3.3%intel@19.1.3.304~ipo+openmp+pic build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
[^]  5fd5vgl      ^ufs-pyenv@1.0.0%intel@19.1.3.304 build_system=bundle arch=linux-sles15-zen2
[^]  y5ruw2g          ^py-cftime@1.0.3.4%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  lzo5wpv              ^py-setuptools@59.4.0%intel@19.1.3.304 build_system=generic arch=linux-sles15-zen2
[^]  w7rkpic              ^py-wheel@0.37.1%intel@19.1.3.304 build_system=generic arch=linux-sles15-zen2
[^]  jayhyji          ^py-cython@0.29.33%intel@19.1.3.304 build_system=python_pip patches=71de066 arch=linux-sles15-zen2
[^]  r3f3n2o          ^py-f90nml@1.4.3%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  chdebgm          ^py-jinja2@3.1.2%intel@19.1.3.304~i18n build_system=python_pip arch=linux-sles15-zen2
[^]  pfbbicd              ^py-markupsafe@2.1.1%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  4h4hbqb          ^py-netcdf4@1.5.3%intel@19.1.3.304~mpi build_system=python_pip patches=255b5ae arch=linux-sles15-zen2
[^]  n3nete7          ^py-numpy@1.22.3%intel@19.1.3.304+blas+lapack build_system=python_pip patches=873745d,e9508c3 arch=linux-sles15-zen2
[^]  xnvkgii              ^openblas@0.3.19%intel@19.1.3.304~bignuma~consistent_fpcsr~ilp64+locking+noavx512+pic+shared build_system=makefile patches=114f95f symbol_suffix=none threads=none arch=linux-sles15-zen2
[^]  ju2llg3          ^py-pandas@1.4.0%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  f6ri2gl              ^py-bottleneck@1.3.5%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  kxmveh2                  ^py-versioneer@0.28%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  fithdis              ^py-numexpr@2.8.3%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  hyzrorv                  ^py-packaging@23.0%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  lwm3lmx                      ^py-flit-core@3.7.1%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  exqvnji              ^py-pytz@2022.2.1%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  gzq5il6          ^py-python-dateutil@2.8.2%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  4a47zrj              ^py-setuptools-scm@7.0.5%intel@19.1.3.304+toml build_system=python_pip arch=linux-sles15-zen2
[^]  6xkt3o3                  ^py-tomli@2.0.1%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  aebkj6y                  ^py-typing-extensions@4.5.0%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  kjrzpvl              ^py-six@1.16.0%intel@19.1.3.304 build_system=python_pip arch=linux-sles15-zen2
[^]  gteogol          ^py-pyyaml@6.0%intel@19.1.3.304+libyaml build_system=python_pip arch=linux-sles15-zen2
[^]  25oabev              ^libyaml@0.2.5%intel@19.1.3.304 build_system=autotools arch=linux-sles15-zen2
[^]  jw353gp      ^w3emc@2.9.2%intel@19.1.3.304~ipo+pic build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-sles15-zen2
ulmononian commented 1 year ago

@mark-a-potts this is running or completed on all machines except azure and s4. do you have s4 access to do this?

on azure, it is not concretizing properly (for some reason it wants to modify basically all the library hashes), and i'm not sure why. since you set up the original 1.4.1 env there, do you mind taking a look (/contrib/EPIC/spack-stack/spack-stack-1.4.1/envs/ufs-pio-2.5.10)?

mark-a-potts commented 1 year ago

@ulmononian I have found that if I can get an install done on one of the PW machines, I can usually take the spack.lock file from that machine and use it directly on the others after a "spack env activate ..." followed by "spack install". On several of them, you do have to run that "module unuse" command to get rid of the fake CRAY paths in the MODULEPATH.

ulmononian commented 1 year ago

@mark-a-potts ok i can give that a try (i.e., take an aws/gcp spack.lock and move to azure). thanks mark!

ulmononian commented 1 year ago

this is done on all major platforms.

@AlexanderRichert-NOAA do you want me to update the modulefiles in ufs-wm 1707 to use this ufs-pio-2.5.10 env?

AlexanderRichert-NOAA commented 1 year ago

Yep let's do it

ulmononian commented 1 year ago

nike'd

climbfuji commented 1 year ago

Is S4 done?

ulmononian commented 1 year ago

Is S4 done?

no. no one had access to the proper space to do the install.

climbfuji commented 1 year ago

Is S4 done?

no. no one had access to the proper space to do the install.

Ok. I can do that Monday if it's still needed.

climbfuji commented 1 year ago

@mark-a-potts @ulmononian @jkbk2004 I created the S4 chained environment with the latest modifications from @AlexanderRichert-NOAA per his email:

Change the spec entry in spack.yaml to:

ufs-weather-model-env%intel~python ^mapl build_type=Release ^esmf fflags="-fp-model precise" cxxflags="-fp-model precise" ^parallelio@2.5.10

Please use module use /data/prod/jedi/spack-stack/spack-stack-1.4.1/envs/ufs-pio-2.5.10/install/modulefiles/Core, the rest is the same as for spack-stack-1.4.1.

Please let me know if this works for you. Thanks!

climbfuji commented 1 year ago

I haven't heard back in this issue, but I think the S4 install is correct. There is a separate ufs-weather-model issue that deals with testing and updating the ufs-w-m code to use those modules.

ulmononian commented 1 year ago

@climbfuji sorry -- i cannot use s4, so i can't check this! @DavidHuber-NOAA did make this comment on the ufs-wm pr: https://github.com/ufs-community/ufs-weather-model/pull/1707#issuecomment-1682225735, which reports that the RTs pass w/ what i think is this stack w/ two minor changes (loading git & nccmp).

ulmononian commented 1 year ago

@climbfuji @AlexanderRichert-NOAA this needs to be done for c5 and hercules since the ufs-wm cm group wants to move on https://github.com/ufs-community/ufs-weather-model/pull/1733 and https://github.com/ufs-community/ufs-weather-model/pull/1784. since i am not sure if these PRs will be added to the WM commit queue before or after rollout of spack-stack 1.5.0, we should probably append the c5/hercules installations to be safe. should i open an issue for this or just go ahead?

AlexanderRichert-NOAA commented 1 year ago

Yeah that makes sense. If you want to reopen this issue and add to the checklist then we'll have a record of it here.

climbfuji commented 1 year ago

I can help with one or both of the platforms, but Hercules is currently down for maintenance.

ulmononian commented 1 year ago

@AlexanderRichert-NOAA cool -- i'll re-open and add these two machines. @climbfuji i am already in the c5 stack space, so i will just do that one today. do you want to run hercules when it's back up? (it's actually the priority from the wm cm perspective i believe).

climbfuji commented 1 year ago

Yes I can do hercules, I am working on that platform these days anyway.

ulmononian commented 1 year ago

gaea c5 done.

climbfuji commented 1 year ago

Hercules is really stubborn. Something is wrong with the spack-stack-1.4.1 code/tree, because the concretizer refuses to work there. I am compiling develop as of today (which has pio 2.5.10 and all the other versions as the ufs-weather-model wants them). Let's just use this for the remainder of the time on Hercules until we switch to spack-stack-1.5.0?

ulmononian commented 1 year ago

sounds like the twelve labors did not end with retrieving cerberus after all. resisting concretization may need to be added as the 13th (@ hesiod).

at this point, this sounds like the best option. thanks for doing this!

p.s.

i will hold hercules RT testing for https://github.com/ufs-community/ufs-weather-model/pull/1733 until this new stack you are installing is ready (fyi @zach1221).

climbfuji commented 1 year ago

I do have a new environment with pio 2.5.10, but I realized that the ESMF and MAPL mods (basically it's only ESMF, but ...) were not in develop yet. So I need to create a chained env off of the just created unified-env. I'll name it ufs-pio-2.5.10 for consistency. It's worth remembering that this is ahead of spack-stack-1.4.1, therefore we shouldn't be surprised if we see b4b differences on other platforms when we move to 1.5.0 that don't show up on Hercules.

climbfuji commented 1 year ago

@AlexanderRichert-NOAA I will note that -fp-model is only an intel option, therefore won't set this for gcc.

climbfuji commented 1 year ago

@ulmononian Hercules finally done:

The usual stuff for both Intel/GNU:

module purge
module use /work/noaa/epic/role-epic/spack-stack/hercules/modulefiles
module load ecflow/5.8.4
module load mysql/8.0.31

For Intel:

module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-dev-20230825/envs/ufs-pio-2.5.10/install/modulefiles/Core
module load stack-intel/2021.9.0
module load stack-intel-oneapi-mpi/2021.9.0
module load stack-python/3.9.14
module load ufs-weather-model-env/1.0.0

For GNU:

module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-dev-20230825/envs/ufs-pio-2.5.10/install/modulefiles/Core
module load stack-gcc/11.3.1
module load stack-openmpi/4.1.5
module load stack-python/3.9.14
module load ufs-weather-model-env/1.0.0

Please give this a try and let me know. Thanks!

climbfuji commented 1 year ago

@ulmononian Can we close this as completed?

ulmononian commented 1 year ago

@ulmononian Can we close this as completed?

yes!

climbfuji commented 1 year ago

Yay. Thanks!