search

SCOREC / particle_structures

unstructured mesh particle data structure
BSD 3-Clause "New" or "Revised" License
3 stars 0 forks source link

GCC 7.4 ICE on summit with Kokkos/develop #15

Open cwsmith opened 5 years ago

cwsmith commented 5 years ago

On Summit using GCC 7.4, cuda/10.1.168, and Kokkos develop (version 3.0), there is an ICE (internal compiler error) while compiling particle_structures/test/typeTest.cpp in the SCS parallel_for macro.

particle_structures 7749497 builds successfully against Kokkos master (2983b80) using GCC 7.4 and cuda/10.1.168 on Summit.

environment

$ module li

Currently Loaded Modules:
  1) hsi/5.0.2.p5   2) xalt/1.1.4   3) lsf-tools/2.0   4) darshan-runtime/3.1.7   5) DefApps   6) tmux/2.2   7) gcc/7.4.0   8) spectrum-mpi/10.3.0.1-20190611   9) cuda/10.1.168  10) cmake/3.15.2

repo hashes

kokkos: 9597d0c (develop branch) particle_structures: aa06763

cmake commands

kokkos

cmake /autofs/nccs-svm1_proj/phy122/cws/develop/kokkos \
-DCMAKE_CXX_COMPILER=/autofs/nccs-svm1_proj/phy122/cws/develop/kokkos/bin/nvcc_wrapper \ 
-DKokkos_ARCH_VOLTA70=ON -DKokkos_ENABLE_SERIAL=ON \
-DKokkos_ENABLE_CUDA=ON -DKokkos_ENABLE_CUDA_LAMBDA=ON \
-DKokkos_ENABLE_DEBUG=off -DKokkos_ENABLE_PROFILING=off \
-DCMAKE_INSTALL_PREFIX=/autofs/nccs-svm1_proj/phy122/cws/develop/build-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168/install

particle_structures

cmake ../particle_structures \
-DCMAKE_CXX_COMPILER=mpiCC \
-DENABLE_KOKKOS=ON \
-DPS_ENABLE_DEBUG_SYMBOLS=ON \
-DPS_ENABLE_OPT=ON \
-DCMAKE_INSTALL_PREFIX=/autofs/nccs-svm1_proj/phy122/cws/develop/build-ps-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168-debugOff/install

compilation command and error

cd /autofs/nccs-svm1_proj/phy122/cws/develop/build-ps-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168-debugOff/test && /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-7.4.0/spectrum-mpi-10.3.0.1-20190611-mto4jwjeylfm6xdlb5yhaphpgnyzcvh7/bin/mpiCC  -DFP64 -DKOKKOS_ENABLED -DSCS_USE_CUDA -I/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support -I/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/algorithms -isystem /autofs/nccs-svm1_proj/phy122/cws/develop/build-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168/install/include  -g -O3   -expt-extended-lambda -arch=sm_70 -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -o CMakeFiles/typeTest.dir/typeTest.cpp.o -c /autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/test/typeTest.cpp
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h: In instantiation of 'particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string)::<lambda(const member_type&)>::<lambda(particle_structs::lid_t&)>::<lambda(particle_structs::lid_t&)> [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; particle_structs::lid_t = int]':
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1155:4:   required from 'struct particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string)::<lambda(const member_type&)>::<lambda(particle_structs::lid_t&)> [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; particle_structs::lid_t = int]::<lambda(particle_structs::lid_t&)>'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1152:21:   required from 'particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string)::<lambda(const member_type&)>::<lambda(particle_structs::lid_t&)> [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; particle_structs::lid_t = int]'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1155:4:   required from 'struct particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string)::<lambda(const member_type&)> [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; Kokkos::Impl::TeamPolicyInternal<Kokkos::Cuda, Kokkos::Cuda>::member_type = Kokkos::Impl::CudaTeamMember]::<lambda(particle_structs::lid_t&)>'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1149:21:   required from 'particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string)::<lambda(const member_type&)> [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; Kokkos::Impl::TeamPolicyInternal<Kokkos::Cuda, Kokkos::Cuda>::member_type = Kokkos::Impl::CudaTeamMember]'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1155:4:   required from 'struct particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string) [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; std::__cxx11::string = std::__cxx11::basic_string<char>]::<lambda(const member_type&)>'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1144:624:   required from 'void particle_structs::SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType&, std::__cxx11::string) [with FunctionType = __nv_dl_wrapper_t<__nv_dl_tag<int (*)(int, char**), main, 1>, particle_structs::Segment<int, Kokkos::Cuda>, particle_structs::Segment<double [2], Kokkos::Cuda> >; DataTypes = particle_structs::MemberTypes<int, double [2]>; ExecSpace = Kokkos::Cuda; std::__cxx11::string = std::__cxx11::basic_string<char>]'
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/test/typeTest.cpp:65:28:   required from here
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1153:40: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1739
         const lid_t particle_id = start+(p*team_size);
                                     ~~~^~~~~~~~~~~~~~
0x102ed143 maybe_undo_parenthesized_ref(tree_node*)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/semantics.c:1738
0x10351713 cp_fold
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/cp-gimplify.c:2180
0x10352737 cp_fold_maybe_rvalue
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/cp-gimplify.c:2042
0x103510e3 cp_fold
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/cp-gimplify.c:2149
0x102c90fb cp_convert_and_check(tree_node*, tree_node*, int)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/cvt.c:640
0x102bac43 cp_build_binary_op(unsigned int, tree_code, tree_node*, tree_node*, int)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/typeck.c:5195
0x101b3c9b build_new_op_1
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/call.c:5978
0x101b484f build_new_op(unsigned int, tree_code, int, tree_node*, tree_node*, tree_node*, tree_node**, int)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/call.c:6022
0x102b0127 build_x_binary_op(unsigned int, tree_code, tree_node*, tree_code, tree_node*, tree_code, tree_node**, int)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/typeck.c:3928
0x102032d3 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:17001
0x10203287 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:16999
0x101f4e73 tsubst_copy_and_build(tree_node*, tree_node*, int, tree_node*, bool, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:16732
0x101f4e73 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:16613
0x101f6997 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:15845
0x101f6997 tsubst_init
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:14532
0x101f5e7f tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:15966
0x101f381b tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:15860
0x101f3a93 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:16090
0x101f381b tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:15860
0x101f3a93 tsubst_expr(tree_node*, tree_node*, int, tree_node*, bool)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/pt.c:16090
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
make[2]: *** [test/CMakeFiles/typeTest.dir/typeTest.cpp.o] Error 1
make[2]: Leaving directory `/autofs/nccs-svm1_proj/phy122/cws/develop/build-ps-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168-debugOff'
make[1]: *** [test/CMakeFiles/typeTest.dir/all] Error 2
make[1]: Leaving directory `/autofs/nccs-svm1_proj/phy122/cws/develop/build-ps-kokkosCudaDbgSerialExec-gcc74-spectrum-cuda101168-debugOff'
make: *** [all] Error 2
cwsmith commented 5 years ago

Moving the product into a variable produces the same error.

git diff
diff --git a/support/SellCSigma.h b/support/SellCSigma.h
index a19170f..022a7cc 100644
--- a/support/SellCSigma.h
+++ b/support/SellCSigma.h
@@ -1150,7 +1150,8 @@ void SellCSigma<DataTypes, ExecSpace>::parallel_for(FunctionType& fn, std::strin
       const lid_t row = slice_to_chunk_cpy(slice) * team_size + slice_row;
       const lid_t element_id = row_to_element_cpy(row);
       Kokkos::parallel_for(Kokkos::ThreadVectorRange(thread, rowLen), [&] (lid_t& p) {
-        const lid_t particle_id = start+(p*team_size);
+        const lid_t pteamsz = p*team_size;
+        const lid_t particle_id = start + pteamsz;
         const lid_t mask = particle_mask_cpy[particle_id];
         (*fn_d)(element_id, particle_id, mask);
       });
autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/test/typeTest.cpp:65:28:   required from here
/autofs/nccs-svm1_proj/phy122/cws/develop/particle_structures/support/SellCSigma.h:1153:25: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1739
         const lid_t pteamsz = p*team_size;
                       ~~^~~~~~~~~~~~~
0x102ed143 maybe_undo_parenthesized_ref(tree_node*)
        /sw/summit/gcc/7.4.0/src/gcc-7.4.0/gcc/cp/semantics.c:1738
diamog commented 5 years ago

The same error exists when building the stack on blockade using gcc/7.3.0-bt47fwr and cuda-10.1