search

ucsf-wynton / wynton-website-hpc

The Official Wynton HPC User Website
https://wynton.ucsf.edu/hpc/
2 stars 14 forks source link

Rocky 8 docs: Claim about modules being older than Rocky 8 software is misleading #129

Closed HenrikBengtsson closed 1 year ago

HenrikBengtsson commented 1 year ago

On https://wynton.ucsf.edu/hpc/software/rocky-8-linux.html it says:

This sounds like a general problem and is possibly misleading too. For example, the CBI modules are up-to-date and provides the latest versions available.

Hopefully, this can be clarified. @ellestad, what is the purpose of those two sentences? What problem is addressed? Can you give an explicit example?

PS. Please don't use upper-case letters for emphasis (e.g. ANY); use _any_ instead. I've fixed this here.

benmwebb commented 1 year ago

Historically, many Wynton software modules were created to provide newer versions of software than those that were available in CentOS 7 Linux

Also not the case for Sali modules, at least. Our stack is generally there to provide software that isn't available on Wynton. There's only a few modules that were intended to provide newer versions of base OS packages (gcc and cmake are the only two that come to mind). We aim for stability though - I'm not spending time constantly updating these packages, unless people explicitly ask for newer versions - so many of them do provide older versions.

ellestad commented 1 year ago

I was just going from my personal experience with a user using modules to load older versions of openMPI and cmake than were available versions on the Rocky 8 Dev Nodes.

>>”Then I module load Sali, mpi/openmpi3-x86_64, and cmake/3.18.3”

I am not sure if the mpi/openmpi3-x86_64 module is supported on Rocky 8, the OpenMPI version installed on Rocky 8 is openmpi-4.1.1.

Also, the locally installed cmake package on Rocky 8 dev nodes is cmake-3.20.2, newer than that module you are loading.
HenrikBengtsson commented 1 year ago

I've added a paragraph on MPI to https://wynton.ucsf.edu/hpc/software/rocky-8-linux.html, which is needed especially since mpi/openmpi-x86_64 gives completely different versions on CentOS 7 and Rocky 8;

  • MPI : OpenMPI is available via the built-in module mpi. On Rocky 8, the default, and only available version is OpenMPI 4.1, which you get via module mpi/openmpi-x86_64 (sic!). Note that, on CentOS 7, mpi/openmpi-x86_64 provided legacy OpenMPI 1.10. On CentOS 7, there was also mpi/openmpi3-x86_64 for OpenMPI 3.1. Because Rocky 8 and CentOS 7 have no versions of OpenMPI in common, it is likely that software tools that were compiled with OpenMPI as a dependency have to be rebuilt for Rocky 8.
HenrikBengtsson commented 1 year ago

cmake/3.18.3

Got it. I've added a paragraph about MPI. Regarding CMake, I think this is a user mistake, because there was no need from the beginning to specify the version; it would have been enough to do module Sali cmake (or module CBI cmake).

I'll see how I can rephrase the current passage for 'Software Repositories' to make it sound less complicated that it actually is. I think most (99.9%) of the modules can be used as before.

ellestad commented 1 year ago

"On Rocky 8, the default, and only available version is OpenMPI 4.1, which you get via module mpi/openmpi-x86_64 (sic!)."

We removed the modulefiles/mpi directory entirely on Rocky 8, so they need to use the locally installed OpenMPI, no module.

ie, from @jlbl:

1) mkdir /wynton/home/opt/config/modulefiles8 2) cp -r /wynton/home/opt/config/modulefiles/* /wynton/home/opt/config/modulefiles8 3) rm -f /wynton/home/opt/config/modulefiles8/mpi [...]

HenrikBengtsson commented 1 year ago

We removed the modulefiles/mpi directory entirely on Rocky 8, ...

My understanding is that /etc/modulefiles/mpi/ is (to be) removed, but Rocky-8-installed /usr/share/modulefiles/mpi/ stays, cf. https://github.com/ucsf-wynton/rocky8-project/issues/17.

Note that the mpi module sets lots of environment variables that you otherwise have to set in /etc/profile.d/, i.e.

hb-test@dev3 ~]$ cat /usr/share/modulefiles/mpi/openmpi-x86_64 
#%Module 1.0
#
#  OpenMPI module for use with 'environment-modules' package:
#
conflict                mpi
prepend-path            PATH            /usr/lib64/openmpi/bin
prepend-path            LD_LIBRARY_PATH /usr/lib64/openmpi/lib
prepend-path            PKG_CONFIG_PATH /usr/lib64/openmpi/lib/pkgconfig
prepend-path            MANPATH         :/usr/share/man/openmpi-x86_64
setenv                  MPI_BIN         /usr/lib64/openmpi/bin
setenv                  MPI_SYSCONFIG   /etc/openmpi-x86_64
setenv                  MPI_FORTRAN_MOD_DIR     /usr/lib64/gfortran/modules/openmpi
setenv                  MPI_INCLUDE     /usr/include/openmpi-x86_64
setenv                  MPI_LIB         /usr/lib64/openmpi/lib
setenv                  MPI_MAN         /usr/share/man/openmpi-x86_64
setenv                  MPI_PYTHON3_SITEARCH    /usr/lib64/python3.6/site-packages/openmpi
setenv                  MPI_COMPILER    openmpi-x86_64
setenv                  MPI_SUFFIX      _openmpi
setenv                  MPI_HOME        /usr/lib64/openmpi

Also, since there are other MPI solutions out there, I think the safest is to have MPI be an "opt-in" to avoid risking MPI_HOME, etc. conflicts.

FWIW, as it is right now, one needs to load the mpi module, e.g.

[hb-test@dev3 ~]$ mpirun --version
-bash: mpirun: command not found
[hb-test@dev3 ~]$ module load mpi
mpi                  mpi/openmpi3-x86_64  mpi/openmpi-x86_64   
[hb-test@dev3 ~]$ module load mpi/openmpi-x86_64
[hb-test@dev3 ~]$ command -v mpirun                                                                                                                                                           
/usr/lib64/openmpi/bin/mpirun
[hb-test@dev3 ~]$ mpirun --version                                                                                                                                                            
mpirun (Open MPI) 4.1.1

Report bugs to http://www.open-mpi.org/community/help/
HenrikBengtsson commented 1 year ago

Closing;

/etc/modulefiles/mpi was removed the other week.

Over simplified the Software Repository paragraph on https://wynton.ucsf.edu/hpc/software/rocky-8-linux.html