search

xsdk-project / xsdk-issues

A repository under which GitHub issues not related to a specific xSDK repo can be filed.
7 stars 0 forks source link

multiple pkg failures with intel-19.1.1.217 #221

Closed balay closed 11 months ago

balay commented 1 year ago

Note: intel compiler is sensitive to gcc in PATH. Currently using:

    modules:
    - gcc/9.5.0/gcc-11.3.1-jpjj2j
#    - gcc/10.4.0/gcc-11.3.1-qlynzz
    - intel-parallel-studio/cluster.2020.1/gcc-11.3.1-isma5w

If I swtich to gcc 10 [or 11] - I get failures in basic pkgs..

pkgs with build failures:

WeiqunZhang commented 1 year ago

The amrex issue seems to be that the Intel compiler used here does not support a C++17 feature (constexpr std::string_view). There is an amrex PR that might provide a workaround. Unfortunately, I don't have access to the particular setup of the environment. @balay Would you be able to test it?

pghysels commented 1 year ago

I tested on Sunspot and Perlmutter and it works (gcc 11.2.0 or 12.1.0 and icc 2021.11.0 or Intel(R) oneAPI DPC++/C++ Compiler 2024.0.0). I don't have access to a system with a gcc that old.

bangerth commented 1 year ago

The deal.II error is the following:

/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/openmpi-4.1.5-ltlwkjkxmido4ho3ytiaj4u6ds2zq6qh/bin/mpic++ -DDEBUG -DKOKKOS_DEPENDENCE -I/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-build-3ul3ju6/source/numerics -I/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-src/source/numerics -I/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-build-3ul3ju6/include -I/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-src/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/netlib-lapack-3.9.1-olfblanh6zprd63ii2rvwpzpnyhh3ak5/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/zlib-ng-2.1.3-ivfac7hcertw3qp4jevlzltm4ri6vhpo/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/boost-1.70.0-u7kg2wtveqga6caxnzwcut36h5eztbnf/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/metis-5.1.0-hfe5jpm72lbqrqlf35giopuxtnx2i6am/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/petsc-3.20.0-4rt7sqnr2i2uuob3qrefpzldantkctza/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/hypre-2.30.0-y7cn3ybs6di4vft2bs5l2u3cpqca2l34/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/superlu-dist-8.2.0-l4lqlzeenhecupue6t6alvpxjmkm2sbu/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/parmetis-4.0.3-j5tqzlznakfuceq3lwuuof7eyfaidazo/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/hdf5-1.12.2-q4lrkwr32b4afcpg73zqz7jmswmavcha/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/kokkos-4.1.00-xofpps3tflv4luukcyqq3zkspxanougg/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/trilinos-14.4.0-hp7xpgy55x55mniozrfgzaxjnvy3zyoj/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/suite-sparse-5.13.0-opxuen4zrmiipdygzd43zlc2wvdptbpu/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/gsl-2.7.1-yfrqdflzwk45w4o23a7a7qc33umvng7u/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/muparser-2.3.4-we4emdlza7obigqsxnrty2fmkp7tqt73/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/oce-0.18.3-4jss2vlf2r4wkqsuteq6im2sujxmk45e/include/oce -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/p4est-2.8-l45qfb3eyxx6h5ovtw7r76r6fvwtcble/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/slepc-3.20.0-ceimvtmggqxog3g5kihmrrmg3ouvoer7/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/sundials-6.6.1-tjdmwzudjq6g7ltcnxn5chkcegbj73fw/include -isystem /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/opt/spack/linux-rocky9-cascadelake/intel-19.1.1.217/intel-tbb-2020.3-5zk5lmksdbd2g2smkfwxbxfhkhoxpuzk/include -std=c++17 -fPIC -w2 -diag-disable=remark -diag-disable=16219 -wd21 -wd68 -wd135 -wd175 -wd177 -wd191 -wd193 -wd279 -wd327 -wd383 -wd854 -wd981 -wd1418 -wd1478 -wd1572 -
wd2259 -wd2536 -wd2651 -wd3415 -wd15531 -wd15552 -wd111 -wd128 -wd185 -wd186 -wd280 -qopenmp-simd -std=c++17 -march=cascadelake -mtune=cascadelake -O0 -g -gdwarf-2 -grecord-gcc-switches -xCORE-AVX512 -DKOKKOS_DEPENDENCE -MD -MT source/numerics/CMakeFiles/object_numerics_debug.dir/data_out_dof_data_inst2.cc.o -MF source/numerics/CMakeFiles/object_numerics_debug.dir/data_out_dof_data_inst2.cc.o.d -o source/numerics/CMakeFiles/object_numerics_debug.dir/data_out_dof_data_inst2.cc.o -c /data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-src/source/numerics/data_out_dof_data_inst2.cc
/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/tuple(553): error: pack "_UElements" does not have the same number of elements as "_Elements"
        __and_<is_nothrow_assignable<_Elements&, _UElements>...>::value;
                                                 ^
          detected during:
            instantiation of "bool std::tuple<_Elements...>::__nothrow_assignable<_UElements...>() [with _Elements=<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>, _UElements=<>]" at line 1011 of "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/type_traits"
            instantiation of class "std::is_assignable<_Tp, _Up> [with _Tp=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> &, _Up=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> &&]" at line 134 of "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/bits/stl_uninitialized.h"
            instantiation of "_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator=std::move_iterator<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *>, _ForwardIterator=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *]" at line 307 of
                      "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/bits/stl_uninitialized.h"
            instantiation of "_ForwardIterator std::__uninitialized_copy_a(_InputIterator, _InputIterator, _ForwardIterator, std::allocator<_Tp> &) [with _InputIterator=std::move_iterator<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *>, _ForwardIterator=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *, _Tp=std::tuple<unsigned int, unsigned int,
                      std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>]" at line 330 of "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/bits/stl_uninitialized.h"
            instantiation of "_ForwardIterator std::__uninitialized_move_if_noexcept_a(_InputIterator, _InputIterator, _ForwardIterator, _Allocator &) [with _InputIterator=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *, _ForwardIterator=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> *, _Allocator=std::allocator<std::tuple<unsigned int, unsigned int,
                      std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>>]" at line 475 of "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/bits/vector.tcc"
            instantiation of "void std::vector<_Tp, _Alloc>::_M_realloc_insert(std::vector<_Tp, _Alloc>::iterator, _Args &&...) [with _Tp=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>, _Alloc=std::allocator<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>>, _Args=<std::tuple<unsigned int &, unsigned int &&, std::string &,
                      dealii::DataComponentInterpretation::DataComponentInterpretation &&>>]" at line 121 of "/apps/spacks/2023-06-07/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/gcc-9.5.0-jpjj2jgpmslw5tt7qtqd4h3zweogatot/include/c++/9.5.0/bits/vector.tcc"
            instantiation of "std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::emplace_back(_Args &&...) [with _Tp=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>, _Alloc=std::allocator<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>>, _Args=<std::tuple<unsigned int &, unsigned int &&, std::string &,
                      dealii::DataComponentInterpretation::DataComponentInterpretation &&>>]" at line 2326 of "/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-src/include/deal.II/numerics/data_out_dof_data.templates.h"
            instantiation of "std::vector<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>, std::allocator<std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>>> dealii::DataOut_DoFData<dim, patch_dim, spacedim, patch_spacedim>::get_nonscalar_data_ranges() const [with dim=2, patch_dim=2, spacedim=2, patch_spacedim=2]" at line 6515 of
                      "/data/xsdkci/VS1mJQ1K/1/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-dealii-9.5.1-3ul3ju6r5vt475jjgsdmopeexquflldi/spack-build-3ul3ju6/source/numerics/data_out_dof_data.inst"

I have to admit that I'm completely stumped by this error. All we do is call std::vector::emplace_back() with a std::tuple type, in this code:

                  ranges.emplace_back(std::forward_as_tuple(
                    output_component,
                    output_component + size - 1,
                    name + "_re",
                    DataComponentInterpretation::component_is_part_of_tensor));

The error happens deep inside the STL. To wit, the function we call is this (template arguments aligned; text taken from the error message):

std::vector<_Tp, _Alloc>::reference
std::vector<_Tp, _Alloc>::emplace_back(_Args &&...)
  [with _Tp=std::tuple<unsigned int,   unsigned int,    std::string,   DataComponentInterpretation::DataComponentInterpretation>,
     _Alloc=...
     _Args=<std::tuple<unsigned int &, unsigned int &&, std::string &, DataComponentInterpretation::DataComponentInterpretation &&>>]

The second to last function before we get to the error is this:

std::is_assignable<_Tp, _Up>
  [with _Tp=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> &,
        _Up=std::tuple<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation> &&]

which apparently calls

bool
std::tuple<_Elements...>::__nothrow_assignable<_UElements...>()
  [with _Elements=<unsigned int, unsigned int, std::string, dealii::DataComponentInterpretation::DataComponentInterpretation>,
        _UElements=<>]

which them reports this error:

pack "_UElements" does not have the same number of elements as "_Elements"
        __and_<is_nothrow_assignable<_Elements&, _UElements>...>::value;

I don't know what to make of this. The only idea I have is that the Intel compiler is not able to fully understand the GCC 9.5 standard library, or that they are in other ways incompatible. We could presumably try to work around this, perhaps by replacing/removing the std::forward_as_tuple call, which I really think is unnecessary here.

Suggestions?

@Rombur @masterleinad

Rombur commented 1 year ago

It looks like datatransferkit, dealii, and strumpack all have the same error message. Googling it, it does look like it's an issue between intel and gcc 9

pghysels commented 1 year ago

We saw this before: https://github.com/xsdk-project/xsdk-issues/issues/184 Then switching back to C++14 instead of C++17 seemed to work. But now strumpack is compiled with -std=c++17, which I think is added by SLATE.

balay commented 1 year ago

There is an amrex PR that might provide a workaround. Unfortunately, I don't have access to the particular setup of the environment. @balay Would you be able to test it?

Yes this fix works! Amrex build goes through with it

==> Installing amrex-23.08-gverk6scnmuizkhmrihunaiycdteeugh [97/100]
==> No binary for amrex-23.08-gverk6scnmuizkhmrihunaiycdteeugh found: installing from source
==> Fetching https://github.com/AMReX-Codes/amrex/releases/download/23.08/amrex-23.08.tar.gz
==> Applied patch /data/balay/spack.y/var/spack/repos/builtin/packages/amrex/3526.diff
==> amrex: Executing phase: 'cmake'
==> amrex: Executing phase: 'build'
==> amrex: Executing phase: 'install'
==> amrex: Successfully installed amrex-23.08-gverk6scnmuizkhmrihunaiycdteeugh
  Stage: 1.57s.  Cmake: 5.08s.  Build: 1m 39.29s.  Install: 0.73s.  Post-install: 0.25s.  Total: 1m 47.30s
balay commented 1 year ago

Googling it, it does look like it's an issue between intel and gcc 9

Its not clear to me how to figure out what the compatible gcc version is for a given intel compiler version. [other than look at spack build/errors - when I try out various gcc versions - with this version of intel compiler]

Then switching back to C++14 instead of C++17 seemed to work.

If C++17 is desired for this release - I can abandon this version of Intel compilers - and look at migrating this test to a newer version.

Rombur commented 1 year ago

If C++17 is desired for this release - I can abandon this version of Intel compilers - and look at migrating this test to a newer version.

I think that's a good idea. DataTransferKit itself does not require C++17 but some of its required dependencies do. Similarly dealii only requires C++14 but it depends on Kokkos which requires C++17 since the 4.0 release.

balay commented 1 year ago 
balay@xsdk:/data/balay/spack.y>nice ./bin/spack install -j24 xsdk@1.0.0 %intel@19.1.1.217 ^trilinos cxxflags=-no-ipo ^openmpi~rsh ^py-numpy@1.22.4
<snip>
alay@xsdk:/data/balay/spack.y>./bin/spack find
-- linux-rocky9-cascadelake / gcc@11.3.1 ------------------------
slate@2023.08.00

-- linux-rocky9-cascadelake / intel@19.1.1.217 ------------------
alquimia@1.1.0                      hiop@1.0.0              openssl@3.1.2            py-pyyaml@6.0
amrex@23.08                         hwloc@2.9.1             p4est@2.8                py-setuptools@63.4.3
arborx@1.4.1                        hypre@2.30.0            parmetis@4.0.3           py-setuptools-scm@7.1.0
arpack-ng@3.9.0                     intel-tbb@2020.3        perl@5.32.1              py-tomli@2.0.1
autoconf@2.69                       kokkos@4.1.00           petsc@3.20.0             py-trove-classifiers@2023.3.9
autoconf-archive@2023.02.20         lapackpp@2023.06.00     pflotran@5.0.0           py-typing-extensions@4.6.3
automake@1.16.2                     libbsd@0.11.7           phist@1.12.1             py-wheel@0.37.1
blaspp@2023.06.00                   libevent@2.1.12         pkgconf@1.7.3            python@3.10.12
blt@0.4.1                           libffi@3.4.4            pmix@4.2.4               raja@0.14.0
boost@1.79.0                        libiconv@1.17           pumi@2.2.8               re2c@2.2
butterflypack@2.3.0                 libmd@1.0.4             py-calver@2022.6.26      readline@8.2
bzip2@1.0.8                         libpciaccess@0.17       py-cython@0.29.36        sed@4.8
ca-certificates-mozilla@2023-05-30  libtool@2.4.6           py-editables@0.3         slepc@3.20.0
camp@0.2.3                          libxcrypt@4.4.35        py-exceptiongroup@1.1.1  sowing@1.1.26-p8
cmake@3.27.4                        libxml2@2.10.3          py-flit-core@3.9.0       sqlite@3.42.0
curl@8.1.2                          libyaml@0.2.5           py-flit-scm@1.7.0        suite-sparse@5.13.0
diffutils@3.7                       m4@1.4.19               py-hatch-vcs@0.3.0       sundials@6.6.1
eigen@3.4.0                         metis@5.1.0             py-hatchling@1.17.0      superlu-dist@8.2.0
exago@1.6.0                         mfem@4.6.0              py-iniconfig@2.0.0       tar@1.34
expat@2.5.0                         mpfr@4.2.0              py-libensemble@1.0.0     tasmanian@8.0
fftw@3.3.10                         muparser@2.3.4          py-mpi4py@3.1.4          texinfo@7.0.3
gdbm@1.23                           ncurses@6.4             py-numpy@1.22.4          trilinos@14.4.0
gettext@0.21.1                      netlib-scalapack@2.2.0  py-packaging@23.1        umpire@6.0.0
ginkgo@1.7.0                        nghttp2@1.52.0          py-pathspec@0.11.1       util-linux-uuid@2.38.1
git@2.39.3                          ninja@1.11.1            py-petsc4py@3.20.0       util-macros@1.19.3
gmake@4.3                           numactl@2.0.14          py-pip@23.1.2            xz@5.2.5
gmp@6.2.1                           oce@0.18.3              py-pluggy@1.0.0          zfp@1.0.0
gsl@2.7.1                           omega-h@scorec.10.6.0   py-psutil@5.9.5          zlib-ng@2.1.3
hdf5@1.14.2                         openblas@0.3.23         py-pydantic@1.10.9       zstd@1.5.5
heffte@2.4.0                        openmpi@4.1.5           py-pytest@7.3.2
==> 120 installed packages

Well kokkos@4.1 (and trilinos@14.4.0) install did go through - with this compiler. Hence assumed (earlier) this (intel compiler) version is usable..

Rombur commented 1 year ago

I don't know about Trilinos but in Kokkos we have a bunch of workarounds to support older Intel versions. For instance we had to introduce a macro because intel 19 has issue with [[deprecated]]. There is no issue with tuple in Kokkos but it's barely used.

balay commented 11 months ago

This build is working now - so closing.

https://gitlab.com/xsdk-project/spack-xsdk/-/jobs/5494592870