Add linking choices for intel-oneapi-mkl

rscohn2 commented 3 years ago

Applications that use mkl need to choose the right set of libraries and command line options as explained by https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl/link-line-advisor.html

The current intel-oneapi-mkl does not provide any compile options and assumes sequential, ilp64. Similar to the intel-mkl package, we need to provide these options as part of the spack package via variants.

I talked to @jasukhar and @NadyaTen and they recommended that we use the link line tool rather than hard code the libraries into the spack package.

We need to decide which variants to provide and whether to use the command line tool/

FYI: @scheibelp @frankwillmore @glennpj

scheibelp commented 3 years ago

I need to think on how to resolve a conflict here:

If I understand correctly: different dependents of intel-oneapi-mkl may request different sets of libraries
- In the terms specified at https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl/link-line-advisor.html that would include choices like static/dynamic linking, threading layer (sequential vs. TBB vs. OpenMP)
Generally variants in Spack are for specifying different installations of a Spack package. If the set of variants differs, that is a new Spack installation. When installing intel-oneapi-mkl, all of these possible library sets are included together (so it would be strange to reinstall the package for a different set of variants).
(This last point is more of a Spack consideration than a statement about intel-oneapi-mkl) Other than the spec, packages don't have a means to coordinate which libraries they link (e.g. they should all probably link the sequential oneAPI MKL libs)

I have an idea on how to resolve that but I need to consult the Spack team internally on that. I'll be able to update this tomorrow. However, @rscohn2 could you let me know if I got something wrong above?

rscohn2 commented 3 years ago

You are correct. It would be good if there was some way to avoid another install when the set of libraries that is used is different.

rscohn2 commented 3 years ago

Other oneapi libraries has the same issue:

https://docs.oneapi.com/versions/latest/onedal/onedal/build_app/build-application.html#applications-on-linux-os

scheibelp commented 3 years ago

@rscohn2 one additional thing I want to confirm: do some of these libraries define the same symbols (in which case it would be wrong to link all of them)?

I spoke internally about this and we decided it needed some more thought, so we will be discussing it again in the next couple days (I anticipate reporting back here by Friday April 2nd). I think we'll be able to address the issue regardless of your answer to the above question (the solution may be simpler if there are no conflicts, but we identified other cases we want to support where there are conflicting libraries).

rscohn2 commented 3 years ago

I believe the libraries define the same symbols.

Some of the MKL options require compile-time options e.g. add -DMKL_ILP64. Is there a way to do that, too?

scheibelp commented 3 years ago

Some of the MKL options require compile-time options e.g. add -DMKL_ILP64. Is there a way to do that, too?

Currently there is not an automatic way to inject compiler flags into a dependent build - i.e. there is no API function we could implement in intel-oneapi-mkl such that all dependents would automatically add -DMKL_ILP64 (so for now, a dependent package would need to check for this and add flags itself). Could you provide more info on when this should happen (e.g. is it when using specific libraries)? We could think on how to provide additional support for it.

rscohn2 commented 3 years ago

DAL and MKL can require some -D and -f options during compilation. For linking, they can need -l and can need start-group and other options. I think it is ok to stick with the current functionality and say dependent packages must add the flags for the complex options. If you are using the virtual blas packages, having spack provide the libraries is a help. If you are using some feature that is specific to the MKL then it is more reasonable to just put it in the dependent package.

rscohn2 commented 3 years ago

Design Doc

Interface

Some of the oneAPI packages support options for choosing the parallelism model and other factors. Threading can be one of TBB, OMP, or sequential. OMP can be GCC-based on icc-based. Packages can use SYCL for offload. A single install of a oneAPI package can support the different models by varying the compile options, link options, and especially the sets of libraries.

oneAPI packages use standard Spack mechanisms to provide paths and libraries so dependent packages can build without knowing the details of library names and paths (e.g. HPL uses MKL as BLAS provider). When there are options, dependent packages can use the Spack spec to control the paths and libraries provided. For example, if a package has virtual dependence on BLAS, then adding hpl ^intel-oneapi-mkl will provide a default implementation via MKL when install hpl. Using the non-default TBB threading would be: hpl ^intel-oneapi-mkl+threading=tbb (may be changed to something else). A dependent package would use self.spec['blas'].libs.ld_flags to get the include paths and libraries required.

Some of the oneAPI configurations are complex and are not easily automated with Spack. The common cases will be covered and dependent packages will use manual configuration for the rest. Assuming the configuration mechanism is variants, these will be supported:

threading={sequential, openmp, tbb}

MKL, DAL, IPP. For openmp threading, use iomp for %intel and %oneapi, otherwise use gomp.

offload=sycl

MKL, DAL, IPP. icx compile requires -fsycl, which must be provided by dependent package. Should work for %gcc with limitations.

Notes:

ilp64

MKL. Only controls libraries. Compiler requires -m64 -DMKL_ILP64, which must be provided by dependent package. Ideally, Spack would provide general support for -m64 and the package would key off that.

fortran95

MKL. Provides BLAS95 & LAPACK95 libraries

Notes:

Some combinations need extra link line options: dt_needed and start_group. Spack cannot provide these automatically so user must configure manually.
MKL cluster interfaces will not be controllable by variants.
Static vs dynamic linking controlled by other mechanisms
I only looked at MKL closely. There might be other variants for the other packages.

Implementation

I looked at link line tool which takes command line options and outputs the link line. Using this would require us to parse the output, separate out the library directories, mkl libraries, system libraries, and compiler runtime libraries, and other link options and handle them appropriately. I feel that it would just replace one problem with another and would rather embed the library names directly in the spack package. We can revisit this in the future as it is not visible to the users of the package. If the info changes, then spack can have version-specific logic to handle it.

scheibelp commented 3 years ago

Some folks on the Spack team met and discussed this today but it proved to be a bit more involved (and we identified possible related scenarios for Cray's libsci and Boost that we also want to possibly address at the same time). We will be meeting again soon (ideally by 4/7) to finish the conversation.

scheibelp commented 3 years ago

MKL, DAL, IPP. icx compile requires -fsycl, which must be provided by dependent package. Should work for %gcc with limitations.

Does this mean that packages which depend on intel-oneapi-mkl and compiling with icx always need to add the -fsycl compiler flag?

NadyaTen commented 3 years ago

actually, no, -fsycl (and few others) are only required in case when customer wants to use OpenMP offload feature.

scheibelp commented 3 years ago

dev-zero commented 2 years ago

When switching from intel-mkl to intel-oneapi-mkl I also noticed that my package (cp2k) failed linking due to missing symbols from scalapack. Indeed it seems that while intel-mkl adds libmkl_scalapack.so to its libs when +mpi is set, intel-oneapi-mkl is missing the mpi variant and does not add them. Same for libmkl_blacs_.... Should I open a separate issue for this or would you add this to the interface list above?

dev-zero commented 2 years ago

... and consistency between intel-oneapi-mkl and intel-oneapi-mpi wrt lp64 vs ilp64 should also be included somehow. Atm intel-oneapi-mpi returns the ilp64 variant in its libs while intel-oneapi-mkl gives the lp64.

rscohn2 commented 2 years ago

I will take a look at MPI and MKL. No need to file a separate issue.

lpoirel commented 1 year ago

The current intel-oneapi-mkl does not provide any compile options and assumes sequential, ilp64. Similar to the intel-mkl package, we need to provide these options as part of the spack package via variants.

All the options discussed here and in https://github.com/spack/seps/pull/2 look wonderful, but might not be available before some time.

However, the current implementation in intel-oneapi-mkl seems like a regression compared to what has been available in intel-mkl.

Would it be possible, until something better is available, to replicate in intel-oneapi-mkl the variant-based mechanics used in intel-mkl ? AFAIU, this would mean porting the blas_libs method (and other similar properties) from IntelPackage to IntelOneApiPackage.

rscohn2 commented 1 year ago

I will look into adding something that matches the way intel-mkl works.

rowanworth commented 7 months ago

I spent hours today trying to get spack 0.21 to link against an external intel-oneapi-mkl build using variant +ilp64 threads=tbb and had no luck at all. Initially I specified this variant in packages.yaml and the concretizer seemed happy with that, however when it got to build time linking against the MKL libraries failed because configure couldn't find the tbb symbols.

Looking closer at the spec I noticed that there was nothing satisfying the tbb virtual, despite this line in intel-oneapi-mkl/package.py:

    depends_on("tbb")

I have a feeling that should be a conditional dependency, but the current state of affairs only helps my use case. Anyway I tried to include tbb explicitly and the concretizer decided that my external mkl was no longer acceptable and tried to go build a new version. Trying to tell it to use my original mkl build caused concretization to fail. Ended up having to compromise by using mkl with threads=none which feels mediocre :/

### initially it finds my external, but tbb is absent
$ spack spec intel-oneapi-mkl threads=tbb
Input spec
--------------------------------
 -   intel-oneapi-mkl threads=tbb

Concretized
--------------------------------
[e]  intel-oneapi-mkl@2021.2.0%gcc@9.2.0~cluster+envmods~ilp64+shared build_system=generic mpi_family=none threads=tbb arch=linux-centos7-x86_64

### specifying a tbb dependency explicitly locates tbb but changes the mkl version
$ spack spec intel-oneapi-mkl threads=tbb ^intel-oneapi-tbb
Input spec
--------------------------------
 -   intel-oneapi-mkl threads=tbb
 -       ^intel-oneapi-tbb

Concretized
--------------------------------
 -   intel-oneapi-mkl@2023.2.0%gcc@9.2.0~cluster+envmods~ilp64+shared build_system=generic mpi_family=none threads=tbb arch=linux-centos7-x86_64
[e]      ^intel-oneapi-tbb@2021.2.0%gcc@9.2.0+envmods build_system=generic arch=linux-centos7-x86_64

### insisting on the already-installed mkl version fails
$ spack spec intel-oneapi-mkl@2021.2.0 threads=tbb ^intel-oneapi-tbb
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    intel-oneapi-mkl@2021.2.0 threads=tbb ^intel-oneapi-tbb is unsatisfiable, errors are:
    internal_error("must choose a single version to satisfy version constraints")
    internal_error("node belongs to no unification set")

rscohn2 commented 7 months ago

When you use an external package, spack ignores the package dependencies. There must be a way to add tbb to the ld_library_path but I cannot investigate until I am back at the office

On Mon, Jan 15, 2024, 7:13 AM Rowan Worth @.***> wrote:

I spent hours today trying to get spack 0.21 to link against an external intel-oneapi-mkl build using variant +ilp64 threads=tbb and had no luck at all. Initially I specified this variant in packages.yaml and the concretizer seemed happy with that, however when it got to build time linking against the MKL libraries failed because configure couldn't find the tbb symbols.

Looking closer at the spec I noticed that there was nothing satisfying the tbb virtual, despite this line in intel-oneapi-mkl/package.py:
depends_on("tbb")
I have a feeling that should be a conditional dependency, but the current state of affairs only helps my use case. Anyway I tried to include tbb explicitly and the concretizer decided that my external mkl was no longer acceptable and tried to go build a new version. Trying to tell it to use my original mkl build caused concretization to fail. Ended up having to compromise by using mkl with threads=none which feels mediocre :/

initially it finds my external, but tbb is absent

$ spack spec intel-oneapi-mkl threads=tbb Input spec

intel-oneapi-mkl threads=tbb

Concretized

[e] @.**@.~cluster+envmods~ilp64+shared build_system=generic mpi_family=none threads=tbb arch=linux-centos7-x86_64

specifying a tbb dependency explicitly locates tbb but changes the mkl version

$ spack spec intel-oneapi-mkl threads=tbb ^intel-oneapi-tbb Input spec

intel-oneapi-mkl threads=tbb

^intel-oneapi-tbb

Concretized

@.**@.~cluster+envmods~ilp64+shared build_system=generic mpi_family=none threads=tbb arch=linux-centos7-x86_64 [e] @.**@.+envmods build_system=generic arch=linux-centos7-x86_64

insisting on the already-installed mkl version fails

$ spack spec @. threads=tbb ^intel-oneapi-tbb ==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message. @. threads=tbb ^intel-oneapi-tbb is unsatisfiable, errors are: internal_error("must choose a single version to satisfy version constraints") internal_error("node belongs to no unification set")

— Reply to this email directly, view it on GitHub https://github.com/spack/spack/issues/22621#issuecomment-1892055575, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOY2WWXMFVGTKFIGYSCV3TYOUMQFAVCNFSM42ADBLC2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBZGIYDKNJVG42Q . You are receiving this because you were mentioned.Message ID: @.***>

spack / spack