kokkos / kokkos-tutorials

Tutorials for the Kokkos C++ Performance Portability Programming Ecosystem
https://kokkos.org
Other
298 stars 100 forks source link

Cannot reproduce openmp example #8

Closed robertsawko closed 6 years ago

robertsawko commented 6 years ago

Hi,

I am trying to learn basics of Kokkos just to see how comfotable I feel with it. I have not particpated in any of your tutorials but I have watched a video on YT and went through several slide decks in this repository.

I am working on P8 machines with Pascals, but also locally on x86+K1100. I installed kokkos myself and I use lmod to manage it. To build kokkos I use something like this:

NVCC_WRAPPER=$(readlink -f kokkos/bin/nvcc_wrapper)
mkdir build
cd build
../kokkos/generate_makefile.bash \
    --prefix=${INSTALL_PATH} \
    --arch="Power8,Pascal60" \
    --compiler=${NVCC_WRAPPER} \
    --with-cuda \
    --with-cuda-options="enable_lambda" \
    --with-openmp \
    && make kokkoslib -j20 && make install

My first problem is that Makefiles in the tutorials overwrite KOKKOS_PATH variables, but I can handle that manually.

My biggest current concern is not being able to run openmp examples e.g. tutorial 01 and 02. If I use your Makefiles I can an error about -arch=60. If I remove all CUDA specific flags I can compile, but I get a CUDA error at runtime

  Total size S = 4194304 N = 4096 M = 1024
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
  In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
  For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
  For unit testing set OMP_PROC_BIND=false
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/rrs59-sxa03/build/kokkos/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:119
Traceback functionality not available

My feeling is that I am doing something terribly wrong due to my misunderstanding. Is it possible to have CUDA and OpenMP backends in one kokkos installation?

mhoemmen commented 6 years ago

It looks like the Makefile might overwrite your KOKKOS_ARCH choice: https://github.com/kokkos/kokkos/blob/d3a941925cbfb71785d8ea68259123ed52d3f9da/example/tutorial/01_hello_world/Makefile#L16

If you set KOKKOS_ARCH in the Makefile, does that fix things?

robertsawko commented 6 years ago

So I can fix things by setting KOKKOS_DEVICES=Cuda,OpenMP and KOKKOS_ARCH=Power8,Pascal60, but my expectation was that if I set OpenMP and Power8, I will get a a host version only.

I also have a version of kokkos compiled only for OpenMP and if load that version then it works as I expect, but I am just surprised I cannot do it with cuda+openmp version.

crtrott commented 6 years ago

Hi, I think the basic issue is that you install Kokkos and than try to build stuff against it. This is not generally how you should do things - in particular the tutorial Makefiles are not setup to work that way.

The main reason is that due to the hardware specific way things are done in the backends you must build applications with practically the same compilers, compiler flags and general settings as the library. So the general recommendation is to build Kokkos inline with the application. Many of our users actually do a thin snapshot of Kokkos into their application repositories and occasionally upgrade that.

One of the straight forward examples where stuff falls apart for example is the question of relocatable device code with CUDA. Both library and application must be build with it either on or off. You can't mix this, and I am not aware of a way of detecting whether you set that flag inside of code.

You can install Kokkos as a library (and for example Trilinos applications do that indirectly through installing Trilinos) but in that case you must pick up stuff like CXXFLAGS and LDFLAGS from the install.

Hope that makes sense.

robertsawko commented 6 years ago

Oh yes, thank you both, @mhoemmen and @crtrott . That does make alot of sense. So basically if I want to test the tutorials, I will have to build a few different versions with backends I am interested in. If I actually develop an application on top of Kokkos or use it through something like Trilinos I will be building my own snapshot.

crtrott commented 6 years ago

Actually if you build the tutorial examples, the Makefile will build the Kokkos library as part of each example. Just set KOKKOS_PATH to whatever your github clone is located. For examples make KOKKOS_PATH=/home/rsawko/github-projects/kokkos KOKKOS_DEVICES=OpenMP KOKKOS_ARCH=Power8 CXX=clang++ Something like that.