search

kokkos / kokkos-remote-spaces

Kokkos Remote Spaces implements distributed Kokkos Views and related APIs for distributed parallel programming.
Other
42 stars 17 forks source link

View construction: build error with MPISpace #86

Closed brian-kelley closed 1 year ago

brian-kelley commented 1 year ago

With Kokkos 4.1, I am able to build and install kokkos-remote-spaces with MPISpace:

cmake \
  -DKRS_ENABLE_MPISPACE=ON \
  -DKokkos_DIR=.../lib64/cmake/Kokkos \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_INSTALL_PREFIX=... \
  -DCMAKE_VERBOSE_MAKEFILE=ON \
  .../kokkos-remote-spaces

but in an application, I get build errors if I try to construct a remote space View by label and extent:

/usr/local/bin/mpicxx -DKOKKOS_DEPENDENCE -DKRS_ENABLE_MPI -DKRS_ENABLE_MPISPACE -isystem /ascldap/users/bmkelle/Questa/MySandbox/scratch/install_mpi/include -isystem /ascldap/users/bmkelle/Questa/MySandbox/scratch/install/include -g -march=core-avx2 -mtune=core-avx2 -std=gnu++17 -MD -MT CMakeFiles/reprod.dir/reprod.cpp.o -MF CMakeFiles/reprod.dir/reprod.cpp.o.d -o CMakeFiles/reprod.dir/reprod.cpp.o -c /ascldap/users/bmkelle/Reprod/reprod.cpp
In file included from /home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_MPISpace.hpp:159,
                 from /home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_RemoteSpaces.hpp:62,
                 from /ascldap/users/bmkelle/Reprod/reprod.cpp:1:
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_MPISpace_DataHandle.hpp: In instantiation of ‘Kokkos::Impl::MPIDataHandle<T, Traits>::MPIDataHandle(const SrcTraits&) [with SrcTraits = double*; T = double; Traits = Kokkos::ViewTraits<double*, Kokkos::Experimental::MPISpace, Kokkos::MemoryTraits<1> >]’:
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_RemoteSpaces_ViewMapping.hpp:1072:23:   required from ‘Kokkos::Impl::ViewMapping<SrcTraits, Kokkos::Experimental::RemoteSpaceSpecializeTag>::ViewMapping(const Kokkos::Impl::ViewCtorProp<Args ...>&, const typename Traits::array_layout&) [with P = {double*}; Traits = Kokkos::ViewTraits<double*, Kokkos::Experimental::MPISpace, Kokkos::MemoryTraits<1> >; typename Traits::array_layout = Kokkos::LayoutRight]’
/home/bmkelle/Questa/MySandbox/scratch/install/include/Kokkos_View.hpp:1442:35:   required from ‘Kokkos::View<DataType, Properties>::View(const Kokkos::Impl::ViewCtorProp<Args ...>&, std::enable_if_t<Kokkos::Impl::ViewCtorProp<Args ...>::has_pointer, typename Kokkos::ViewTraits<DataType, Properties ...>::array_layout>&) [with P = {double*}; DataType = double*; Properties = {Kokkos::Experimental::MPISpace, Kokkos::MemoryTraits<1>}; std::enable_if_t<Kokkos::Impl::ViewCtorProp<Args ...>::has_pointer, typename Kokkos::ViewTraits<DataType, Properties ...>::array_layout> = Kokkos::LayoutRight]’
/home/bmkelle/Questa/MySandbox/scratch/install/include/Kokkos_View.hpp:1579:75:   required from ‘Kokkos::View<DataType, Properties>::View(Kokkos::View<DataType, Properties>::pointer_type, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t) [with DataType = double*; Properties = {Kokkos::Experimental::MPISpace, Kokkos::MemoryTraits<1>}; Kokkos::View<DataType, Properties>::pointer_type = double*; size_t = long unsigned int]’
/home/bmkelle/Questa/MySandbox/scratch/install/include/impl/Kokkos_ViewMapping.hpp:3064:19:   required from ‘std::enable_if_t<(std::is_trivial<Dummy>::value && std::is_trivially_copy_assignable<Dummy>::value)> Kokkos::Impl::ViewValueFunctor<DeviceType, ValueType, true>::construct_shared_allocation() [with Dummy = double; DeviceType = Kokkos::Device<Kokkos::Serial, Kokkos::Experimental::MPISpace>; ValueType = double; std::enable_if_t<(std::is_trivial<Dummy>::value && std::is_trivially_copy_assignable<Dummy>::value)> = void]’
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_RemoteSpaces_ViewMapping.hpp:1214:9:   required from ‘Kokkos::Impl::SharedAllocationRecord<void, void>* Kokkos::Impl::ViewMapping<SrcTraits, Kokkos::Experimental::RemoteSpaceSpecializeTag>::allocate_shared(const Kokkos::Impl::ViewCtorProp<Args ...>&, const typename Traits::array_layout&, bool) [with P = {std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Kokkos::Experimental::MPISpace, Kokkos::Serial}; T = Kokkos::ViewTraits<double*, Kokkos::Experimental::MPISpace>; Traits = Kokkos::ViewTraits<double*, Kokkos::Experimental::MPISpace>; typename Traits::array_layout = Kokkos::LayoutRight]’
/home/bmkelle/Questa/MySandbox/scratch/install/include/Kokkos_View.hpp:1421:45:   required from ‘Kokkos::View<DataType, Properties>::View(const Kokkos::Impl::ViewCtorProp<Args ...>&, std::enable_if_t<(! Kokkos::Impl::ViewCtorProp<Args ...>::has_pointer), typename Kokkos::ViewTraits<DataType, Properties ...>::array_layout>&) [with P = {std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >}; DataType = double*; Properties = {Kokkos::Experimental::MPISpace}; std::enable_if_t<(! Kokkos::Impl::ViewCtorProp<Args ...>::has_pointer), typename Kokkos::ViewTraits<DataType, Properties ...>::array_layout> = Kokkos::LayoutRight]’
/home/bmkelle/Questa/MySandbox/scratch/install/include/Kokkos_View.hpp:1514:75:   required from ‘Kokkos::View<DataType, Properties>::View(const Label&, std::enable_if_t<Kokkos::Impl::is_view_label<Label>::value, const long unsigned int>, size_t, size_t, size_t, size_t, size_t, size_t, size_t) [with Label = char [7]; DataType = double*; Properties = {Kokkos::Experimental::MPISpace}; std::enable_if_t<Kokkos::Impl::is_view_label<Label>::value, const long unsigned int> = const long unsigned int; size_t = long unsigned int]’
/ascldap/users/bmkelle/Reprod/reprod.cpp:31:95:   required from here
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_MPISpace_DataHandle.hpp:44:17: error: request for member ‘ptr’ in ‘arg’, which is of non-class type ‘double* const’
   44 |       : ptr(arg.ptr), win(arg.win), win_offset(arg.win_offset) {}
      |             ~~~~^~~
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_MPISpace_DataHandle.hpp:44:31: error: request for member ‘win’ in ‘arg’, which is of non-class type ‘double* const’
   44 |       : ptr(arg.ptr), win(arg.win), win_offset(arg.win_offset) {}
      |                           ~~~~^~~
/home/bmkelle/Questa/MySandbox/scratch/install_mpi/include/Kokkos_MPISpace_DataHandle.hpp:44:52: error: request for member ‘win_offset’ in ‘arg’, which is of non-class type ‘double* const’
   44 |       : ptr(arg.ptr), win(arg.win), win_offset(arg.win_offset) {}
      |                                                ~~~~^~~~~~~~~~
make[2]: *** [CMakeFiles/reprod.dir/reprod.cpp.o] Error 1
make[2]: Leaving directory `/home/bmkelle/reprodBuild'
make[1]: *** [CMakeFiles/reprod.dir/all] Error 2
make[1]: Leaving directory `/home/bmkelle/reprodBuild'
make: *** [all] Error 2

Here is a complete program that reproduces the issue with MPISpace enabled, but compiles and runs OK when SHMEMSpace is enabled:

#include <Kokkos_RemoteSpaces.hpp>
int main(int argc, char** argv)
{
  int mpi_thread_level_available;
  int mpi_thread_level_required = MPI_THREAD_MULTIPLE;
#ifdef KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_SERIAL
  mpi_thread_level_required = MPI_THREAD_SINGLE;
#endif
  MPI_Init_thread(&argc, &argv, mpi_thread_level_required,
                  &mpi_thread_level_available);
  assert(mpi_thread_level_available >= mpi_thread_level_required);
#ifdef KRS_ENABLE_SHMEMSPACE
  shmem_init_thread(mpi_thread_level_required, &mpi_thread_level_available);
  assert(mpi_thread_level_available >= mpi_thread_level_required);
#endif
  Kokkos::initialize(argc, argv);
  {
    Kokkos::View<double*, Kokkos::Experimental::DefaultRemoteMemorySpace> myView("myView", 100);
  }
  Kokkos::finalize();
#if defined(KRS_ENABLE_SHMEMSPACE)
  shmem_finalize();
#else
  MPI_Finalize();
#endif
  return 0;
}
janciesko commented 1 year ago

Thanks for reporting. I was able to reproduce. This issue has been resolved in v1.0.