Error executing setup-meta-modules

[rjdave]

Describe the bug Building the devel branch of the stack (as of 05/22/2024) with Intel then GCC results in a "Multiple version matches" error when it tries to create the meta modules.

To Reproduce Note: I have forced the use of Open MPI 4.1.6 by requesting it in site/packages.yaml to facilitate using TotalView (the 5.x line has removed the MPIR process acquisition interface required by TotalView).

Build the stack with both Intel and GCC compilers then try to execute spack stack setup-meta-modules in either environment and you will get output similar to below:

$ spack stack setup-meta-modules
Configuring basic directory information ...
  ... script directory: /opt/src/spack-stack/spack-ext/lib/jcsda-emc/spack-stack/stack
  ... base directory: /opt/src/spack-stack/spack-ext/lib/jcsda-emc/spack-stack
  ... spack directory: /opt/src/spack-stack/spack
Configuring active spack environment ...
  ... environment directory: /opt/src/spack-stack/envs/devel20240523_intel
Parsing spack environment main config ...
  ... install directory: /opt/spack-stack
Parsing spack environment modules config ...
  ... configured to use lmod modules
  ... module directory: /opt/spack-stack/modulefiles
Parsing spack environment package config ...
  ... list of possible compilers: '['intel@2021.8.0', 'gcc', 'clang', 'oneapi', 'xl', 'nag', 'fj', 'aocc']'
  ... list of possible mpi providers: '['openmpi@4.1.6', 'openmpi', 'mpich']'
['intel', 'openmpi', 'gcc']
 ... stack compilers: '{'intel': ['2021.8.0'], 'gcc': ['11.4.1']}'
==> Error: Multiple version matches for openmpi@4.1.6 in /opt/spack-stack/modulefiles/openmpi: ['4.1.6-2edufp4', '4.1.6-6gupjjp']

Curiously, If I temporarily move 4.1.6-2edufp4, for example, and run the spack stack setup-meta-modules it gets further (far enough to make the stack loadable), but still errors like below so there are no stack-openmpi or stack-python modules created.

$ spack stack setup-meta-modules
Configuring basic directory information ...
  ... script directory: /opt/src/spack-stack/spack-ext/lib/jcsda-emc/spack-stack/stack
  ... base directory: /opt/src/spack-stack/spack-ext/lib/jcsda-emc/spack-stack
  ... spack directory: /opt/src/spack-stack/spack
Configuring active spack environment ...
  ... environment directory: /opt/src/spack-stack/envs/devel20240523_intel
Parsing spack environment main config ...
  ... install directory: /opt/spack-stack
Parsing spack environment modules config ...
  ... configured to use lmod modules
  ... module directory: /opt/spack-stack/modulefiles
Parsing spack environment package config ...
  ... list of possible compilers: '['intel@2021.8.0', 'gcc', 'clang', 'oneapi', 'xl', 'nag', 'fj', 'aocc']'
  ... list of possible mpi providers: '['openmpi@4.1.6', 'openmpi', 'mpich']'
['intel', 'openmpi', 'gcc']
 ... stack compilers: '{'intel': ['2021.8.0'], 'gcc': ['11.4.1']}'
 ... stack mpi providers: '{'openmpi': {'4.1.6-6gupjjp': {'intel': ['2021.8.0']}}}'
  ... core compilers: ['gcc@4.6']
Preparing meta module directory ...
  ... meta module directory : /opt/spack-stack/modulefiles/Core
Creating compiler modules ...
  ... configuring stack compiler gcc@11.4.1
  ... ... CC  : /usr/bin/gcc
  ... ... CXX : /usr/bin/g++
  ... ... F77 : /usr/bin/gfortran
  ... ... FC' : /usr/bin/gfortran
  ... ... COMPFLAGS:
  ... ... MODULELOADS:
  ... ... MODULEPREREQS:
  ... ... MODULEPATH  : /opt/spack-stack/modulefiles/gcc/11.4.1
  ... writing /opt/spack-stack/modulefiles/Core/stack-gcc/11.4.1.lua
  ... configuring stack compiler intel@2021.8.0
  ... ... CC  : /opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/icc
  ... ... CXX : /opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/icpc
  ... ... F77 : /opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/ifort
  ... ... FC' : /opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/bin/intel64/ifort
  ... ... COMPFLAGS: setenv("CFLAGS", "-diag-disable=10441")
setenv("CXXFLAGS", "-diag-disable=10441")
setenv("FFLAGS", "-diag-disable=10441")
  ... ... MODULELOADS: load("intel/2021.8.0")
  ... ... MODULEPREREQS: prereq("intel/2021.8.0")
  ... ... ENVVARS  : prepend_path("LD_LIBRARY_PATH", "/opt/sw/apps/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin")
  ... ... MODULEPATH  : /opt/spack-stack/modulefiles/intel/2021.8.0
  ... writing /opt/spack-stack/modulefiles/Core/stack-intel/2021.8.0.lua
==> Error: Package with matching name is incompatible: ordereddict([('version', ['4.1.6']), ('variants', '~internal-hwloc +two_level_namespace')])

Expected behavior spack stack setup-meta-modules should complete without errors

System: Linux Workstation running Rocky 9. Intel is 2023.0.0 (2021.8.0 legacy compilers (icc,icpc,ifort)) and GCC is 11.4.1 system package manager install.

Additional context This appears to be somewhat recent and is possibly due to the fact that spack now adds the shortened hash to the openmpi (and presumably other MPI implementations) version sub-directories. In previous versions, /path/to/spack-stack/modulefiles/openmpi would contain version number (i.e. 4.1.6) folders. However, now the hash is added to the folder name so instead of /path/to/spack-stack/modulefiles/openmpi containing a single 4.1.6 folder with intel and gcc sub-directories, there is a 4.1.6-<intel_built_hash> and a 4.1.6-<gcc_built_hash> folder.

[climbfuji]

This problem occurs when you use the same MPI provider (openmpi-4.1.6 in your case) with different compilers?

[rjdave]

This problem occurs when you use the same MPI provider (openmpi-4.1.6 in your case) with different compilers?

Yes. I also had a similar error with spack-stack earlier this year when Open MPI first switched to 5.x. I had previously built the stack with the Intel compilers when Open MPI was still at 4.1.6. A couple months later I pulled the latest stack which had upgraded Open MPI to 5.0 to build with the GCC 11.4 compilers. In that case, the spack stack setup-meta-modules command produced the same error complaining that there were multiple version choices for Open MPI. However, when I temporarily moved /path/to/spack-stack/modulefiles/openmpi/4.1.6 and ran the command again, it completed successfully. I moved the 4.1.6 directory back into place and was able to use stack-intel and stack-gcc without issue.