Build failure with `intel-mpi`.

[1uc]

The following steps fail to build on BB5:

spack env create libsonata_with_intel_mpi
spack env activate -p libsonata_with_intel_mpi
spack add libsonata^intel-mpi
spack install -j16

The resulting config file is:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [libsonata, intel-mpi]
  view: true
  concretizer:
    unify: true

The error is:

1 error found in build log:
     17    -- HIGHFIVE 2.7.1: (Re)Detecting Highfive dependencies (HIGHFIVE_USE_INSTALL_DEPS=NO)
     18    -- Found HDF5: hdf5-shared (found version "1.14.0")
     19    -- Found Boost: /gpfs/bbp.cscs.ch/ssd/apps/bsd/2023-02-23/stage_applications/install_gcc-12.2.0-skylake/boost-1.80.0-dcp2kq/include (found version 
           "1.80.0") found components: system serialization
     20    -- Found MPI_C: /gpfs/bbp.cscs.ch/home/groshein/spack_install/software/install_gcc-12.2.0-skylake/intel-mpi-2019.10.317-cau5lw/compilers_and_librar
           ies_2020.4.317/linux/mpi/intel64/bin/mpicc (found version "3.1")
     21    -- Found MPI_CXX: /gpfs/bbp.cscs.ch/home/groshein/spack_install/software/install_gcc-12.2.0-skylake/intel-mpi-2019.10.317-cau5lw/compilers_and_libr
           aries_2020.4.317/linux/mpi/intel64/bin/mpicxx (found version "3.1")
     22    -- Found MPI: TRUE (found version "3.1")
  >> 23    CMake Error at CMakeLists.txt:80 (find_package):
     24      By not providing "Findfmt.cmake" in CMAKE_MODULE_PATH this project has
     25      asked CMake to find a package configuration file provided by "fmt", but
     26      CMake did not find one.
     27    
     28      Could not find a package configuration file provided by "fmt" with any of
     29      the following names:

See build log for details:
  /tmp/groshein/spack-stage/spack-stage-libsonata-0.1.23-gtpa7aftkd55dmrpojatq5nissz23q5n/spack-build-out.txt

Finally the spack spec:

$ spack spec
Input spec
--------------------------------
libsonata

Concretized
--------------------------------
libsonata@0.1.23%gcc@12.2.0~ipo+mpi~tests build_system=cmake build_type=RelWithDebInfo cxxstd=14 arch=linux-rhel7-skylake
    ^cmake@3.24.3%gcc@12.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rhel7-skylake
    ^fmt@9.1.0%gcc@12.2.0~ipo+pic~shared build_system=cmake build_type=RelWithDebInfo cxxstd=11 patches=08fb707 arch=linux-rhel7-skylake
    ^highfive@2.7.1%gcc@12.2.0+boost~eigen~ipo+mpi~page_buffer_patch~xtensor build_system=cmake build_type=RelWithDebInfo arch=linux-rhel7-skylake
        ^boost@1.80.0%gcc@12.2.0~atomic~chrono~clanglibcpp~container~context~contract~coroutine~date_time~debug~exception~fiber+filesystem~graph~graph_parallel~icu~iostreams~json~locale~log~math~mpi+multithreaded~nowide~numpy+pic~program_options~python~random~regex+serialization+shared~signals~singlethreaded~stacktrace+system~taggedlayout+test~thread~timer~type_erasure~versionedlayout~wave build_system=generic cxxstd=98 patches=a440f96 visibility=hidden arch=linux-rhel7-skylake
        ^hdf5@1.14.0%gcc@12.2.0+cxx~fortran+hl~ipo~java~map+mpi~page_buffer_patch+shared~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo patches=0b5dd6f arch=linux-rhel7-skylake
            ^pkgconf@1.8.0%gcc@12.2.0 build_system=autotools arch=linux-rhel7-skylake
            ^zlib@1.2.13%gcc@12.2.0+optimize+pic+shared build_system=makefile arch=linux-rhel7-skylake
    ^intel-mpi@2019.10.317%gcc@12.2.0~external-libfabric build_system=generic arch=linux-rhel7-skylake
        ^cpio@2.13%gcc@12.2.0 build_system=autotools arch=linux-rhel7-skylake
    ^nlohmann-json@3.9.1%gcc@12.2.0~ipo+multiple_headers build_system=cmake build_type=RelWithDebInfo arch=linux-rhel7-skylake
    ^py-setuptools-scm@7.0.5%gcc@12.2.0+toml build_system=python_pip arch=linux-rhel7-skylake
        ^py-packaging@21.3%gcc@12.2.0 build_system=python_pip arch=linux-rhel7-skylake
            ^py-pyparsing@3.0.9%gcc@12.2.0 build_system=python_pip arch=linux-rhel7-skylake
        ^py-pip@22.2.2%gcc@12.2.0 build_system=generic arch=linux-rhel7-skylake
        ^py-setuptools@63.0.0%gcc@12.2.0 build_system=generic arch=linux-rhel7-skylake
        ^py-tomli@2.0.1%gcc@12.2.0 build_system=python_pip arch=linux-rhel7-skylake
        ^py-typing-extensions@4.3.0%gcc@12.2.0 build_system=python_pip arch=linux-rhel7-skylake
            ^py-flit-core@3.7.1%gcc@12.2.0 build_system=python_pip arch=linux-rhel7-skylake
        ^py-wheel@0.37.1%gcc@12.2.0 build_system=generic arch=linux-rhel7-skylake
        ^python@3.10.8%gcc@12.2.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-rhel7-skylake

Input spec
--------------------------------
intel-mpi

Concretized
--------------------------------
intel-mpi@2019.10.317%gcc@12.2.0~external-libfabric build_system=generic arch=linux-rhel7-skylake
    ^cpio@2.13%gcc@12.2.0 build_system=autotools arch=linux-rhel7-skylake

[mgeplf]

Weird that it has:

^fmt@9.1.0%gcc@12.2.0~ipo+pic~shared build_system=cmake build_type=RelWithDebInfo cxxstd=11 patches=08fb707 arch=linux-rhel7-skylake

But doesn't find the Findfmt.cmake. Any ideas @matz-e, before I look into this?

[mgeplf]

The libsonata/package.py has:

depends_on("fmt@4.0:")

Which I thought would imply build, maybe it doesn't? Not sure what MPI does to this whole thing, there are a bunch of conditionals on MPI on the package.py, too...

[mgeplf]

I tried setting depends_on("fmt@4.0:", type="build",) and got the same error.

In the build log; I see this for both:

-DCMAKE_PREFIX_PATH:STRING=[....];/gpfs/bbp.cscs.ch/ssd/apps/bsd/2023-02-23/stage_applications/install_gcc-12.2.0-skylake/fmt-9.1.0-t7yyhx;[....]

Which looks right, and the ordering of the pre and post [...] are the same...very confusing.

[matz-e]

Fixed by copying libfabric* into the same directory as libmpi*. Now something that's less than a band-aid...

[matz-e]

Alternative: Intel® stuff is outdated, we're supposed to use 1api™. libsonata^intel-oneapi-mpi compiles out-of-the-box for me.

[mgeplf]

Note to self, @matz-e found this by opening the spack-build-out.txt; which pointed to CMakeError.log, which had:

/gpfs/bbp.cscs.ch/ssd/apps/bsd/2023-02-23/stage_compilers/install_gcc-4.8.5-haswell/binutils-2.37-l5h4u2/bin/ld: warning: libfabric.so.1, needed by /gpfs/bbp.cscs.ch/project/proj30/tickets/NSETM-2061-minis-validatio
n-dask-jobqueue/install/install_gcc-12.2.0-skylake/intel-mpi-2019.10.317-cau5lw/compilers_and_libraries_2020.4.317/linux/mpi/intel64/lib/release/libmpi.so, not found (try using -rpath or -rpath-link)                
/gpfs/bbp.cscs.ch/ssd/apps/bsd/2023-02-23/stage_compilers/install_gcc-4.8.5-haswell/binutils-2.37-l5h4u2/bin/ld: /gpfs/bbp.cscs.ch/project/proj30/tickets/NSETM-2061-minis-validation-dask-jobqueue/install/install_gcc
-12.2.0-skylake/intel-mpi-2019.10.317-cau5lw/compilers_and_libraries_2020.4.317/linux/mpi/intel64/lib/release/libmpi.so: undefined reference to `fi_freeinfo@FABRIC_1.1' 
[....]

[mgeplf]

I think this can be closed; seems to be on the spack side.