Ifpack2 build failure with std=c++14 with CUDA-10.1 and CUDA-10.2

[finkandreas]

Trying to build with cuda-10.1 or cuda-10.2 and setting Kokkos_CXX_STANDARD=14 fails the build in Ifpack2.

I use this cmake command:

cmake -DCMAKE_BUILD_TYPE=Release \
           -DCMAKE_VERBOSE_MAKEFILE=ON \
           -DCMAKE_CXX_COMPILER=/tmp/trilinos.src/packages/kokkos/bin/nvcc_wrapper \
           -DTrilinos_ENABLE_Ifpack2=ON \
           -DKokkos_ENABLE_CUDA=ON \
           -DKokkos_ENABLE_CUDA_UVM=ON \
           -DKokkos_ENABLE_CUDA_LAMBDA=ON \
           -DTPL_ENABLE_CUDA=ON \
           -DTPL_ENABLE_CUSPARSE=ON \
           -DKokkos_CXX_STANDARD=14 \
           -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
           ../trilinos.src/

The build fails with the following error message:

cd /tmp/trilinos.build/packages/ifpack2/src && /tmp/trilinos.src/packages/kokkos/bin/nvcc_wrapper   -I/tmp/trilinos.build -I/tmp/trilinos.src/packages/ifpack2/src -I/tmp/trilinos.build/packages/ifpack2/src -I/tmp/trilinos.src -I/tmp/trilinos.build/packages/amesos2/src -I/tmp/trilinos.src/packages/amesos2/src -I/tmp/trilinos.src/packages/amesos2/src/KLU2/Include -I/tmp/trilinos.src/packages/amesos2/src/KLU2/Source -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/AMD/Include -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/COLAMD/Include -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/BTF/Include -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/UFconfig -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/CAMD/Include -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/CCOLAMD/Include -I/tmp/trilinos.src/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Include -I/tmp/trilinos.src/packages/tpetra/core/ext -I/tmp/trilinos.build/packages/tpetra/core/ext -I/tmp/trilinos.src/packages/tpetra/core/inout -I/tmp/trilinos.build/packages/tpetra/core/inout -I/tmp/trilinos.src/packages/tpetra/core/src -I/tmp/trilinos.src/packages/tpetra/core/src/kokkos_refactor -I/tmp/trilinos.build/packages/tpetra/core/src -I/tmp/trilinos.src/packages/tpetra/tsqr/src -I/tmp/trilinos.build/packages/tpetra/tsqr/src -I/tmp/trilinos.src/packages/tpetra/classic/LinAlg -I/tmp/trilinos.src/packages/tpetra/classic/NodeAPI -I/tmp/trilinos.build/packages/tpetra/classic/NodeAPI -I/tmp/trilinos.build/packages/tpetra/classic/src -I/tmp/trilinos.src/packages/tpetra/classic/src -I/tmp/trilinos.build/packages/teuchos/kokkoscomm/src -I/tmp/trilinos.src/packages/teuchos/kokkoscomm/src -I/tmp/trilinos.build/packages/teuchos/kokkoscompat/src -I/tmp/trilinos.src/packages/teuchos/kokkoscompat/src -I/tmp/trilinos.src/packages/teuchos/parameterlist/src -I/tmp/trilinos.src/packages/teuchos/parser/src -I/tmp/trilinos.build/packages/teuchos/core/src -I/tmp/trilinos.src/packages/teuchos/core/src -I/tmp/trilinos.build/packages/kokkos/core/src -I/tmp/trilinos.src/packages/kokkos/core/src -I/tmp/trilinos.build/packages/kokkos -I/usr/local/cuda/include -I/tmp/trilinos.src/packages/teuchos/comm/src -I/tmp/trilinos.src/packages/teuchos/remainder/src -I/tmp/trilinos.build/packages/teuchos/remainder/src -I/tmp/trilinos.src/packages/teuchos/numerics/src -I/tmp/trilinos.build/packages/kokkos-kernels/src -I/tmp/trilinos.src/packages/kokkos-kernels/src -I/tmp/trilinos.src/packages/kokkos-kernels/src/impl -I/tmp/trilinos.src/packages/kokkos-kernels/src/impl/tpls -I/tmp/trilinos.src/packages/kokkos-kernels/src/blas -I/tmp/trilinos.src/packages/kokkos-kernels/src/blas/impl -I/tmp/trilinos.src/packages/kokkos-kernels/src/sparse -I/tmp/trilinos.src/packages/kokkos-kernels/src/sparse/impl -I/tmp/trilinos.src/packages/kokkos-kernels/src/graph -I/tmp/trilinos.src/packages/kokkos-kernels/src/graph/impl -I/tmp/trilinos.src/packages/kokkos-kernels/src/batched -I/tmp/trilinos.src/packages/kokkos-kernels/src/common -I/tmp/trilinos.build/packages/kokkos/algorithms/src -I/tmp/trilinos.src/packages/kokkos/algorithms/src -I/tmp/trilinos.build/packages/kokkos/containers/src -I/tmp/trilinos.src/packages/kokkos/containers/src -I/tmp/trilinos.build/packages/epetra/src -I/tmp/trilinos.src/packages/epetra/src -I/tmp/trilinos.build/packages/epetraext/src -I/tmp/trilinos.src/packages/epetraext/src -I/tmp/trilinos.src/packages/epetraext/src/transform -I/tmp/trilinos.src/packages/epetraext/src/inout -I/tmp/trilinos.src/packages/epetraext/src/coloring -I/tmp/trilinos.src/packages/epetraext/src/model_evaluator -I/tmp/trilinos.src/packages/epetraext/src/block -I/tmp/trilinos.src/packages/epetraext/src/restrict -I/tmp/trilinos.build/packages/triutils/src -I/tmp/trilinos.src/packages/triutils/src -I/tmp/trilinos.build/packages/shylu/shylu_node/hts/src -I/tmp/trilinos.src/packages/shylu/shylu_node/hts/src -I/tmp/trilinos.build/packages/belos/xpetra/src -I/tmp/trilinos.src/packages/belos/xpetra/src -I/tmp/trilinos.build/packages/belos/tpetra/src -I/tmp/trilinos.src/packages/belos/tpetra/src -I/tmp/trilinos.build/packages/belos/epetra/src -I/tmp/trilinos.src/packages/belos/epetra/src -I/tmp/trilinos.build/packages/belos/src -I/tmp/trilinos.src/packages/belos/src -I/tmp/trilinos.build/packages/aztecoo/src -I/tmp/trilinos.src/packages/aztecoo/src -I/tmp/trilinos.src/packages/xpetra/sup/Matrix -I/tmp/trilinos.src/packages/xpetra/sup/StridedMap -I/tmp/trilinos.src/packages/xpetra/sup/Utils -I/tmp/trilinos.build/packages/xpetra/sup -I/tmp/trilinos.src/packages/xpetra/src/CrsGraph -I/tmp/trilinos.src/packages/xpetra/src/CrsMatrix -I/tmp/trilinos.src/packages/xpetra/src/BlockedCrsMatrix -I/tmp/trilinos.src/packages/xpetra/src/DistObject -I/tmp/trilinos.src/packages/xpetra/src/Export -I/tmp/trilinos.src/packages/xpetra/src/Headers -I/tmp/trilinos.src/packages/xpetra/src/Import -I/tmp/trilinos.src/packages/xpetra/src/Map -I/tmp/trilinos.src/packages/xpetra/src/BlockedMap -I/tmp/trilinos.src/packages/xpetra/src/MultiVector -I/tmp/trilinos.src/packages/xpetra/src/BlockedMultiVector -I/tmp/trilinos.src/packages/xpetra/src/BlockedVector -I/tmp/trilinos.src/packages/xpetra/src/Operator -I/tmp/trilinos.src/packages/xpetra/src/Platform -I/tmp/trilinos.src/packages/xpetra/src/RowGraph -I/tmp/trilinos.src/packages/xpetra/src/RowMatrix -I/tmp/trilinos.src/packages/xpetra/src/Utils -I/tmp/trilinos.src/packages/xpetra/src/Utils/ForwardDeclaration -I/tmp/trilinos.src/packages/xpetra/src/Vector -I/tmp/trilinos.build/packages/xpetra/src -I/tmp/trilinos.src/packages/thyra/adapters/tpetra/src -I/tmp/trilinos.src/packages/thyra/adapters/epetra/src -I/tmp/trilinos.src/packages/thyra/core/src -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/operator_vector/fundamental -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/operator_vector/extended -I/tmp/trilinos.src/packages/thyra/core/src/support/operator_vector/client_support -I/tmp/trilinos.src/packages/thyra/core/src/support/operator_vector/adapter_support -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/operator_solve/fundamental -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/operator_solve/extended -I/tmp/trilinos.src/packages/thyra/core/src/support/operator_solve/client_support -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/nonlinear/model_evaluator/fundamental -I/tmp/trilinos.src/packages/thyra/core/src/support/nonlinear/model_evaluator/client_support -I/tmp/trilinos.src/packages/thyra/core/src/interfaces/nonlinear/solvers/fundamental -I/tmp/trilinos.src/packages/thyra/core/src/support/nonlinear/solvers/client_support -I/tmp/trilinos.build/packages/thyra/core/src -I/tmp/trilinos.src/packages/thyra/core/example/operator_vector -I/tmp/trilinos.src/packages/rtop/src -I/tmp/trilinos.src/packages/rtop/src/interfaces -I/tmp/trilinos.src/packages/rtop/src/support -I/tmp/trilinos.src/packages/rtop/src/ops_lib -I/tmp/trilinos.src/packages/rtop/src/lapack -I/tmp/trilinos.build/packages/rtop/src -I/tmp/trilinos.src/packages/thyra/adapters/epetraext/src/model_evaluator -I/tmp/trilinos.src/packages/thyra/adapters/epetraext/src/transformer -I/tmp/trilinos.src/packages/zoltan2/src -I/tmp/trilinos.build/packages/zoltan2/src -I/tmp/trilinos.src/packages/zoltan2/src/algorithms -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/partition -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/order -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/color -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/match -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/map -I/tmp/trilinos.src/packages/zoltan2/src/algorithms/zoltan -I/tmp/trilinos.src/packages/zoltan2/src/environment -I/tmp/trilinos.src/packages/zoltan2/src/include -I/tmp/trilinos.src/packages/zoltan2/src/input -I/tmp/trilinos.src/packages/zoltan2/src/models -I/tmp/trilinos.src/packages/zoltan2/src/problems -I/tmp/trilinos.src/packages/zoltan2/src/util -I/tmp/trilinos.src/packages/zoltan2/src/directory -I/tmp/trilinos.build/packages/zoltan/src -I/tmp/trilinos.src/packages/zoltan/src/include -I/tmp/trilinos.src/packages/zoltan/src/all -I/tmp/trilinos.src/packages/zoltan/src/coloring -I/tmp/trilinos.src/packages/zoltan/src/graph -I/tmp/trilinos.src/packages/zoltan/src/ha -I/tmp/trilinos.src/packages/zoltan/src/hier -I/tmp/trilinos.src/packages/zoltan/src/hsfc -I/tmp/trilinos.src/packages/zoltan/src/lb -I/tmp/trilinos.src/packages/zoltan/src/matrix -I/tmp/trilinos.src/packages/zoltan/src/order -I/tmp/trilinos.src/packages/zoltan/src/par -I/tmp/trilinos.src/packages/zoltan/src/params -I/tmp/trilinos.src/packages/zoltan/src/tpls -I/tmp/trilinos.src/packages/zoltan/src/phg -I/tmp/trilinos.src/packages/zoltan/src/rcb -I/tmp/trilinos.src/packages/zoltan/src/reftree -I/tmp/trilinos.src/packages/zoltan/src/simple -I/tmp/trilinos.src/packages/zoltan/src/timer -I/tmp/trilinos.src/packages/zoltan/src/Utilities/Communication -I/tmp/trilinos.src/packages/zoltan/src/Utilities/DDirectory -I/tmp/trilinos.src/packages/zoltan/src/Utilities/Timer -I/tmp/trilinos.src/packages/zoltan/src/Utilities/shared -I/tmp/trilinos.src/packages/zoltan/src/zz -I/tmp/trilinos.src/packages/zoltan/siMPI/pyMPI/siMPI  -pedantic -Wall -Wno-long-long -Wwrite-strings   -std=c++14  -expt-extended-lambda  -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -O3 -DNDEBUG   -o CMakeFiles/ifpack2.dir/Ifpack2_ReorderFilter_Cuda.cpp.o -c /tmp/trilinos.build/packages/ifpack2/src/Ifpack2_ReorderFilter_Cuda.cpp
/tmp/trilinos.src/packages/tpetra/core/src/Tpetra_withLocalAccess.hpp(248): error: class "Tpetra::Details::GetNonowningLocalObject<<error-type>>" has no member "nonowning_local_object_type"
          detected during:
            instantiation of type "Tpetra::with_local_access_function_argument_type<<error-type>>" 
(882): here
            instantiation of "void Tpetra::Details::WithLocalAccess<FirstLocalAccessType, Rest...>::withLocalAccess(Tpetra::Details::WithLocalAccess<FirstLocalAccessType, Rest...>::current_user_function_type, FirstLocalAccessType, Rest...) [with FirstLocalAccessType=Tpetra::Details::LocalAccess<Tpetra::Vector<Tpetra::Details::DefaultTypes::scalar_type, Tpetra::Details::DefaultTypes::local_ordinal_type, Tpetra::Details::DefaultTypes::global_ordinal_type, Tpetra::Details::DefaultTypes::node_type>, Kokkos::HostSpace, Tpetra::Details::Access<Tpetra::Details::EAccess::ReadWrite>>, Rest=<>]" 
(933): here
            instantiation of "void Tpetra::withLocalAccess(Tpetra::Details::ArgsToFunction<LocalAccessTypes...>::type, LocalAccessTypes...) [with LocalAccessTypes=<Tpetra::Details::LocalAccess<Tpetra::Vector<Tpetra::Details::DefaultTypes::scalar_type, Tpetra::Details::DefaultTypes::local_ordinal_type, Tpetra::Details::DefaultTypes::global_ordinal_type, Tpetra::Details::DefaultTypes::node_type>, Kokkos::HostSpace, Tpetra::Details::Access<Tpetra::Details::EAccess::ReadWrite>>>]" 
/tmp/trilinos.src/packages/ifpack2/src/Ifpack2_Relaxation_def.hpp(1080): here
            instantiation of "void Ifpack2::Relaxation<MatrixType>::compute() [with MatrixType=Tpetra::RowMatrix<double, int, longlong, Kokkos_Compat_KokkosCudaWrapperNode>]" 
/tmp/trilinos.build/packages/ifpack2/src/Ifpack2_Relaxation_Cuda.cpp(61): here

1 error detected in the compilation of "/tmp/tmpxft_00005b82_00000000-6_Ifpack2_Relaxation_Cuda.cpp1.ii".
packages/ifpack2/src/CMakeFiles/ifpack2.dir/build.make:593: recipe for target 'packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_Relaxation_Cuda.cpp.o' failed
make[2]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/Ifpack2_Relaxation_Cuda.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/trilinos.build'
make[1]: *** [packages/ifpack2/src/CMakeFiles/ifpack2.dir/all] Error 2
CMakeFiles/Makefile2:7197: recipe for target 'packages/ifpack2/src/CMakeFiles/ifpack2.dir/all' failed
make[1]: Leaving directory '/tmp/trilinos.build'
Makefile:165: recipe for target 'all' failed
make: *** [all] Error 2

In my opinion this is a bug in nvcc, but maybe there is a workaround. Host compiler is GCC-8.3.0. A reproducer can be seen with docker and the following Dockerfile:

FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04
MAINTAINER Andreas Fink <finkandreas@web.de>

# Update all packages and install the development toolset (newer version of GNU gcc and tools)
RUN apt-get update \
 && apt-get upgrade -y \
 && apt-get install -y libblas-dev liblapack-dev wget zlib1g-dev python-dev cmake libmpich-dev git libbison-dev flex rsync nano python3-dev g++-8 \
 && rm -Rf /var/lib/apt/lists/* \
 && for i in gcc g++ ; do \
        update-alternatives --install /usr/bin/${i} ${i} /usr/bin/${i}-7 10 ; \
        update-alternatives --install /usr/bin/${i} ${i} /usr/bin/${i}-8 20 ; \
        update-alternatives --set ${i} /usr/bin/${i}-8 ; \
    done

RUN git clone https://github.com/trilinos/Trilinos.git /tmp/trilinos.src \
  && mkdir /tmp/trilinos.build \
  && cd /tmp/trilinos.build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
           -DCMAKE_VERBOSE_MAKEFILE=ON \
           -DCMAKE_CXX_COMPILER=/tmp/trilinos.src/packages/kokkos/bin/nvcc_wrapper \
           -DTrilinos_ENABLE_Ifpack2=ON \
           -DKokkos_ENABLE_CUDA=ON \
           -DKokkos_ENABLE_CUDA_UVM=ON \
           -DKokkos_ENABLE_CUDA_LAMBDA=ON \
           -DTPL_ENABLE_CUDA=ON \
           -DTPL_ENABLE_CUSPARSE=ON \
           -DKokkos_CXX_STANDARD=14 \
           -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
           ../trilinos.src/ \
  && make -j4

[cgcgcg]

@trilinos/ifpack2

[mhoemmen]

@finkandreas Thanks for reporting! The only reason I need rw_type and lcl_vec_type is because I don't have C++14 generic lambdas. I don't have a build set up like this, so could you please do an experiment for me? Delete lines 1052-1057 of Ifpack2_Relaxation_def.hpp, that is, the following lines:

      using rw_type =
        decltype (readWrite (gblDiag).on (Kokkos::HostSpace ()));
      // Once we have C++14, we can get rid of this alias and use
      // "auto" in the lambda below.
      using lcl_vec_type =
        Tpetra::with_local_access_function_argument_type<rw_type>;

replace line 1061 with the following:

        ([&A_row, L1_eta, numMyRows] (const auto& diag) {

and rebuild. The only reason I didn't do this is because Trilinos can't use C++14 things, but if this works for you, then we can use this as a work-around (protected with a macro).

[finkandreas]

I tried your suggestion, but the build error is still the same.

[mhoemmen]

@finkandreas Thanks for trying! What happens if you use Kokkos_CXX_STANDARD=17?

[finkandreas]

that's not a valid standard for nvcc (at least not in cuda-10.2)

[mhoemmen]

@finkandreas Thanks for trying! : - ) FYI, it looks like Trilinos now has a non-CUDA C++14 build: https://github.com/trilinos/Trilinos/pull/6969

[jwillenbring]

@srajama1 Any suggestion where we can go with this?

[jwillenbring]

CC: @kddevin due to Tpetra reference.

[vbrunini]

We started seeing this error with intel builds when we turned c++14 on for sierra on Friday as well. I've got a workaround for it and a few other issues that popped up that I'll put a PR up for today.

[jwillenbring]

Thanks @vbrunini !

[vbrunini]

See #7767

[kddevin]

7767 should work around this compiler problem. I will close this issue.

@jwillenbring we probably need a C++14 CUDA PR test.