Trilinos build failure in CI - with Intel compilers [with MKL]

[balay]

https://gitlab.com/xsdk-project/spack-xsdk/-/jobs/6133623710

$ nice python3.9 ./bin/spack install xsdk@1.0.0%oneapi@2023.2.0 ^intel-oneapi-mkl ^intel-oneapi-mpi

6 errors found in build log:
     1379    
     1380    
     1381    Finished configuring Trilinos!
     1382    
     1383    -- If publishing results using Trilinos, please cite us: https://t
             rilinos.github.io/cite.html
     1384    -- Configuring done (14.5s)
  >> 1385    CMake Error in packages/teuchos/numerics/src/CMakeLists.txt:
     1386      Imported target "BLAS::all_libs" includes non-existent path
     1387    
     1388        "/software/intel/oneapi/mkl/2023.2.0/include/mkl"
     1389    
     1390      in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
     1391    
     ...
     1395    
     1396      * The installation package was faulty and references files it do
             es not
     1397      provide.
     1398    
     1399    
     1400    

spack-build-out.txt

similarly https://gitlab.com/xsdk-project/spack-xsdk/-/jobs/6133623713

$ ./bin/spack install -j12 xsdk@1.0.0~dealii%oneapi@2022.2.0 ^intel-oneapi-mpi ^intel-oneapi-mkl ^netlib-scalapack%gcc@9.4.0 ^butterflypack~openmp ^py-numpy@1.25.2

[balay]

commit 43a94e981ac223d235ae1afbf55de8f0aead4c66 (origin/xsdk-1.0.0)
Author: Satish Balay <balay@mcs.anl.gov>
Date:   Tue Nov 14 11:00:19 2023 -0600

    xsdk: add version 1.0.0 (#40825)

This old snapshot build fine - so the failure is with current spack/develop.

Using git bisect with:

xsdk@pj01:~/spack.x$ ./bin/spack install trilinos@14.4.0%oneapi@2023.2.0 ^intel-oneapi-mkl ^intel-oneapi-mpi

gives:

3d744e7c951eab4fa70b95c15c5cabc969dd47b3 is the first bad commit
commit 3d744e7c951eab4fa70b95c15c5cabc969dd47b3
Author: Andrey Perestoronin <andrey.perestoronin@intel.com>
Date:   Mon Nov 27 18:13:04 2023 +0000

    intel-oneapi 2024.0.0: added new version to packages (#41135)

    * oneapi 2024.0.0 release

    * oneapi v2 directory support and some cleanups

    * sycl abi change requires 2024 compilers for packages that use sycl

    ---------

    Co-authored-by: Robert Cohn <robert.s.cohn@intel.com>

 .../docs/build_systems/inteloneapipackage.rst      | 25 +++++--
 lib/spack/spack/build_systems/oneapi.py            | 53 ++++++++------
 lib/spack/spack/compilers/oneapi.py                | 12 +++
 .../cloud_pipelines/stacks/e4s-oneapi/spack.yaml   |  3 +
 .../packages/intel-oneapi-advisor/package.py       | 10 +++
 .../builtin/packages/intel-oneapi-ccl/package.py   | 10 +++
 .../intel-oneapi-compilers-classic/package.py      |  4 +-
 .../packages/intel-oneapi-compilers/package.py     | 85 ++++++++++++++++------
 .../builtin/packages/intel-oneapi-dal/package.py   | 10 +++
 .../builtin/packages/intel-oneapi-dnn/package.py   | 23 +++++-
 .../builtin/packages/intel-oneapi-dpct/package.py  | 10 +++
 .../builtin/packages/intel-oneapi-dpl/package.py   | 20 +++--
 .../packages/intel-oneapi-inspector/package.py     | 10 +++
 .../builtin/packages/intel-oneapi-ipp/package.py   | 10 +++
 .../builtin/packages/intel-oneapi-ippcp/package.py | 10 +++
 .../builtin/packages/intel-oneapi-itac/package.py  | 10 +++
 .../builtin/packages/intel-oneapi-mkl/package.py   | 24 ++++--
 .../builtin/packages/intel-oneapi-mpi/package.py   | 24 +++++-
 .../builtin/packages/intel-oneapi-tbb/package.py   | 16 ++++
 .../builtin/packages/intel-oneapi-vpl/package.py   |  6 ++
 .../builtin/packages/intel-oneapi-vtune/package.py | 10 +++
 21 files changed, 313 insertions(+), 72 deletions(-)

[balay]

This build is with pre-installed Intel OneAPI

compilers.yaml

- compiler:
    spec: oneapi@=2023.2.0
    paths:
      cc: /software/intel/oneapi/compiler/2023.2.0/linux/bin/icx
      cxx: /software/intel/oneapi/compiler/2023.2.0/linux/bin/icpx
      f77: /software/intel/oneapi/compiler/2023.2.0/linux/bin/ifx
      fc: /software/intel/oneapi/compiler/2023.2.0/linux/bin/ifx
    flags: {}
    operating_system: fedora39
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []

packages.yaml

  intel-oneapi-mkl:
    buildable: false
    externals:
    - spec: intel-oneapi-mkl@2023.2.0%oneapi@2023.2.0
      prefix: /software/intel/oneapi
  intel-oneapi-mpi:
    buildable: false
    externals:
    - spec: intel-oneapi-mpi@2021.10.0%oneapi@2023.2.0
      prefix: /software/intel/oneapi
  intel-oneapi-dpl:
    buildable: false
    externals:
    - spec: intel-oneapi-dpl@2022.2.0%oneapi@2023.2.0
      prefix: /software/intel/oneapi

So I let spack install oneapi-mkl - and that works [so can't used pre-installed MKL anymore - or it requires a different spec in packages.yaml now?]

[balay]

cc: @eugeneswalker for any insights [wrt MKL usage here.]

[eugeneswalker]

We need this fix just merged:

• https://github.com/spack/spack/pull/42614

[balay]

We need this fix just merged:

Great! Works for me. Thanks!