compilation problem with lammps-flare for gpu via kokkos

[eric-yu-zhu]

Dear flare team,

I am having problem compiling lammps-flare with gpu via kokkos. On the other hand, I can run flare and compile lammps with gpu via kokkos without flare. So the problem might be in the connetion between flare and lammps. Any help would be appreciated!

Version info

• flare: 1.3.0 (cloned from the development branch, git 2ec05b3)
• lammps: lammps-stable_23Jun2022_update1
• gpu: Nvidia - RTX A6000
• cuda: 11.4
• operating system: Ubuntu 20.04.4 LTS

Command history

• unzip flare_lammps/lammps-stable_23Jun2022_update1.zip -d .
• mv lammps-stable_23Jun2022_update1 lammps_gpu
• cd flare_v2/lammps_plugins
• ./install.sh ../../lammps_gpu
• cd ../../lammps_gpu
• rm -rf build && mkdir build && cd build
• cmake ../cmake -DPKG_MACHDYN=yes -DDOWNLOAD_EIGEN3=yes -DPKG_MANYBODY=yes -DKokkos_ARCH_HOSTARCH=yes -DKokkos_ARCH_GPUARCH=yes -DKokkos_ENABLE_CUDA=yes -DKokkos_ENABLE_OPENMP=yes -DPKG_KOKKOS=yes
• make -j8

Error message

[ 13%] Building CXX object _deps/kokkoskernels-build/src/CMakeFiles/kokkoskernels.dir/impl/tpls/KokkosBlas_Rocm_tpl.cpp.o
[ 13%] Building CXX object _deps/kokkoskernels-build/src/CMakeFiles/kokkoskernels.dir/impl/tpls/KokkosKernels_tpl_handles.cpp.o
   /home/eric/UserData/user2022b/proj01_NanoMD2/lammps_gpu/build/_deps/kokkoskernels-src/src/common/Kokkos_ArithTraits.hpp(979): error: no instance of function template "Kokkos::abs" matches the argument list  argument types are: (std::enable_if_t<true, float>)
   /home/eric/UserData/user2022b/proj01_NanoMD2/lammps_gpu/build/_deps/kokkoskernels-src/src/common/Kokkos_ArithTraits.hpp(998): error: no instance of overloaded function "Kokkos::pow" matches the argument list  argument types are: (std::enable_if_t<true, float>, std::enable_if_t<true, float>)
   /home/eric/UserData/user2022b/proj01_NanoMD2/lammps_gpu/build/_deps/kokkoskernels-src/src/common/Kokkos_ArithTraits.hpp(1003): error: no instance of function template "Kokkos::sqrt" matches the argument list  argument types are: (std::enable_if_t<true, float>)
   /home/eric/UserData/user2022b/proj01_NanoMD2/lammps_gpu/build/_deps/kokkoskernels-src/src/common/Kokkos_ArithTraits.hpp(1007): error: namespace "Kokkos" has no member "cbrt"
   /home/eric/UserData/user2022b/proj01_NanoMD2/lammps_gpu/build/_deps/kokkoskernels-src/src/common/Kokkos_ArithTraits.hpp(1011): error: no instance of overloaded function "Kokkos::exp" matches the argument list  argument types are: (std::enable_if_t<true, float>)

[YuuuXie]

This seems to be a kokkos version issue? @anjohan

[anjohan]

Hi @eric-yu-zhu ,

I think this is a mismatch between the Kokkos library bundled with LAMMPS and the KokkosKernels that we download.

In lammps/cmake/CMakeLists.txt, locate kokkoskernels near the bottom and add the GIT_TAG line below:

        FetchContent_Declare(
            kokkoskernels
            GIT_REPOSITORY https://github.com/kokkos/kokkos-kernels.git
            GIT_TAG 3.6.00
        )

Btw, I wouldn't necessarily expect

-DKokkos_ENABLE_CUDA=yes -DKokkos_ENABLE_OPENMP=yes

to work together. While I did template PairFLAREKokkos on the device, there is a decent chance that I have forgotten to use it in some places, and I have never tried running Kokkos with OpenMP in a CUDA build.

If you find problems running OpenMP, you can make a separate build directory without CUDA enabled. As of writing, the Kokkos pair style is also optimized for GPUs only, so it will probably be faster to just run the non-Kokkos version with MPI when not using the GPU.

[eric-yu-zhu]

Hi anjohan, thank you so much for your advice. The solution works nicely!

I believe you are right about the -DKokkos_ENABLE_OPENMP flag. It was copied from lammps website.

For those who have trouble with kokkos like me, here's the recipe that worked on my gpu workstation

compilation recipe

• unzip flare_lammps/lammps-stable_23Jun2022_update1.zip -d .
• mv lammps-stable_23Jun2022_update1 lammps_gpu_v2
• cd flare_v2/lammps_plugins
• ./install.sh ../../lammps_gpu_v2

• edit lammps_gpu_v2/cmake/CMakeLists.txt and replace if(PKG_KOKKOS) ... endif() block if(PKG_KOKKOS) set(KokkosKernels_ADD_DEFAULT_ETI OFF CACHE BOOL "faster build") set(KokkosKernels_ENABLED_COMPONENTS BLAS CACHE STRING "faster build")

  if(EXTERNAL_KOKKOS)
      find_package(KokkosKernels REQUIRED)
  else()
      include(FetchContent)
      FetchContent_Declare(
          kokkoskernels
          GIT_REPOSITORY https://github.com/kokkos/kokkos-kernels.git
          GIT_TAG 3.6.00
      )
      FetchContent_MakeAvailable(kokkoskernels)
  endif()
  target_link_libraries(lammps PUBLIC Kokkos::kokkoskernels)

  endif()

• cd ../../lammps_gpu_v2
• rm -rf build && mkdir build && cd build
• cmake ../cmake -DPKG_MACHDYN=yes -DDOWNLOAD_EIGEN3=yes -DPKG_MANYBODY=yes -DKokkos_ARCH_HOSTARCH=yes -DKokkos_ARCH_GPUARCH=yes -DKokkos_ENABLE_CUDA=yes -DPKG_KOKKOS=yes
• make -j8

[anjohan]

Great!

I'll have a look to see if there's a way to automatically determine the KokkosKernels version based on the LAMMPS+Kokkos versions selected by the user.