keichi / kEDM

A high-performance implementation of Empirical Dynamic Modeling (EDM)
https://kedm.readthedocs.io/
MIT License
13 stars 5 forks source link

error while installing using CMake #40

Open davidgwyrick opened 1 year ago

davidgwyrick commented 1 year ago

Hello, I seem to be getting a compile error during installation related to Cuda. My workstation has a GeForce RTX 4090 with cuda 12.1 installed. I first created a conda environment, which I installed the cuda toolbox. It fails to find the correct nvidia GPU architecture even though I specified the correct one (I think). any ideas? Thank you in advance.

(kedm) david.wyrick@robinson:~/Git/kEDM$ cmake -B build -S . -DKEDM_ENABLE_PYTHON=ON -DKEDM_ENABLE_GPU=ON -DKokkos_ARCH_ADA89=ON -DKEDM_ENABLE_EXECUTABLES=ON -- Setting default Kokkos CXX standard to 17 -- The project name is: Kokkos -- Using internal gtest for testing -- Compiler Version: 12.2.128 -- kokkos_launch_compiler (/home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/bin/kokkos_launch_compiler) is enabled... -- Using -std=c++17 for C++17 standard as feature -- CUDA auto-detection of architecture failed with /usr/bin/c++. Enabling CUDA language ONLY to auto-detect architecture... CMake Error at src/thirdparty/kokkos/cmake/kokkos_arch.cmake:801 (MESSAGE): CUDA enabled but no NVIDIA GPU architecture currently enabled and auto-detection failed. Please give one -DKokkosARCH{..}=ON' to enable an NVIDIA GPU architecture.

You can yourself try to compile /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/cmake/compile_tests/cuda_compute_capability.cc and run the executable. If you are cross-compiling, you should try to do this on a compute node. Call Stack (most recent call first): src/thirdparty/kokkos/cmake/kokkos_tribits.cmake:244 (INCLUDE) src/thirdparty/kokkos/CMakeLists.txt:213 (KOKKOS_SETUP_BUILD_ENVIRONMENT)

-- Built-in Execution Spaces: -- Device Parallel: Kokkos::Cuda -- Host Parallel: Kokkos::OpenMP -- Host Serial: NONE

-- Architectures: -- Using internal desul_atomics copy -- Kokkos Devices: OPENMP;CUDA, Kokkos Backends: OPENMP;CUDA -- Boost.Math: standalone mode ON -- pybind11 v2.9.2 -- kEDM build configuration: -- C++ Compiler: GNU 11.4.0 -- /usr/bin/c++ -- CPU backend: ON -- GPU backend: ON -- Executables: ON -- Python bindings: ON -- MPI: OFF -- Unit tests: ON -- LIKWID: OFF -- Scratch memory: ON -- SIMD primitives: ON -- Configuring incomplete, errors occurred! See also "/home/david.wyrick/Git/kEDM/build/CMakeFiles/CMakeOutput.log".

davidgwyrick commented 1 year ago

Ok, i changed the GPU architecture to be AMPERE86 because supposedly lovelace (ADA89) is back compatible. but I think there is still issues with some of the ctests and python tests. For the python tests, it can't even read in kEDM.

`(kedm2) david.wyrick@robinson:~/Git/kEDM$ cmake -B build -S . -DKEDM_ENABLE_PYTHON=ON -DKEDM_ENABLE_GPU=ON -DKokkos_ARCH_AMPERE86=ON -DKEDM_ENABLE_EXECUTABLES=ON -- The CXX compiler identification is GNU 11.4.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Setting default Kokkos CXX standard to 17 -- The project name is: Kokkos -- Using internal gtest for testing -- Configured git information in /home/david.wyrick/Git/kEDM/build/generated/Kokkos_Version_Info.cpp -- Compiler Version: 11.7.64 -- kokkos_launch_compiler (/home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/bin/kokkos_launch_compiler) is enabled... -- Using -std=c++17 for C++17 standard as feature -- Built-in Execution Spaces: -- Device Parallel: Kokkos::Cuda -- Host Parallel: Kokkos::OpenMP -- Host Serial: NONE

-- Architectures: -- AMPERE86 -- Found CUDAToolkit: /home/david.wyrick/anaconda3/envs/kedm2/include (found version "11.7.64") -- Looking for C++ include pthread.h -- Looking for C++ include pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- Found TPLCUDA: TRUE -- Found TPLLIBDL: /usr/include -- Found OpenMP_CXX: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") -- Using internal desul_atomics copy -- Kokkos Devices: OPENMP;CUDA, Kokkos Backends: OPENMP;CUDA -- The C compiler identification is GNU 11.4.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Boost.Math: standalone mode ON -- Found CUDA: /home/david.wyrick/anaconda3/envs/kedm2 (found version "11.7") -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libcurl.so;/usr/lib/x86_64-linux-gnu/libpthread.a;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.a;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.7") -- Setting build type to 'RelWithDebInfo' as none was specified. -- pybind11 v2.9.2 -- Found PythonInterp: /home/david.wyrick/anaconda3/envs/kedm2/bin/python (found version "3.8.17") -- Found PythonLibs: /home/david.wyrick/anaconda3/envs/kedm2/lib/libpython3.8.so -- Performing Test HAS_FLTO -- Performing Test HAS_FLTO - Success -- kEDM build configuration: -- C++ Compiler: GNU 11.4.0 -- /usr/bin/c++ -- CPU backend: ON -- GPU backend: ON -- Executables: ON -- Python bindings: ON -- MPI: OFF -- Unit tests: ON -- LIKWID: OFF -- Scratch memory: ON -- SIMD primitives: ON -- Configuring done CMake Warning at CMakeLists.txt:89 (add_executable): Cannot generate a safe runtime search path for target edm-xmap because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly.

CMake Warning at CMakeLists.txt:93 (add_executable): Cannot generate a safe runtime search path for target knn-bench because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly.

CMake Warning at CMakeLists.txt:96 (add_executable): Cannot generate a safe runtime search path for target lookup-bench because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly.

CMake Warning at CMakeLists.txt:99 (add_executable): Cannot generate a safe runtime search path for target smap-bench because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly.

CMake Warning at src/thirdparty/pybind11/tools/pybind11Tools.cmake:165 (add_library): Cannot generate a safe runtime search path for target _kedm because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly. Call Stack (most recent call first): CMakeLists.txt:136 (pybind11_add_module)

CMake Warning at CMakeLists.txt:151 (add_executable): Cannot generate a safe runtime search path for target kedm-test because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libcrypto.so.3] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib
runtime library [libz.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm2/lib

Some of these libraries may not be found correctly.

-- Generating done -- Build files have been written to: /home/david.wyrick/Git/kEDM/build (kedm2) david.wyrick@robinson:~/Git/kEDM$ cmake --build build [ 1%] Building CXX object src/thirdparty/kokkos/simd/src/CMakeFiles/kokkossimd.dir/Kokkos_SIMD_dummy.cpp.o [ 3%] Linking CXX static library libkokkossimd.a [ 3%] Built target kokkossimd [ 5%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o [ 7%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Command_Line_Parsing.cpp.o [ 8%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o [ 10%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Error.cpp.o [ 12%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_ExecPolicy.cpp.o [ 14%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostBarrier.cpp.o [ 15%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostSpace.cpp.o [ 17%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostSpace_deepcopy.cpp.o [ 19%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_HostThreadTeam.cpp.o [ 21%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_MemoryPool.cpp.o [ 22%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_MemorySpace.cpp.o [ 24%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Profiling.cpp.o /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/core/src/impl/Kokkos_Profiling.cpp: In function ‘Kokkos::Tools::Impl::InitializationStatus Kokkos::Tools::Impl::parse_environment_variables(Kokkos::Tools::InitArguments&)’: /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/core/src/impl/Kokkos_Profiling.cpp:179:81: warning: missing initializer for member ‘Kokkos::Tools::Impl::InitializationStatus::error_message’ [-Wmissing-field-initializers] 179 return { ^ /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/core/src/impl/Kokkos_Profiling.cpp: In function ‘Kokkos::Tools::Impl::InitializationStatus Kokkos::Tools::Impl::initialize_tools_subsystem(const Kokkos::Tools::InitArguments&)’: /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/core/src/impl/Kokkos_Profiling.cpp:195:65: warning: missing initializer for member ‘Kokkos::Tools::Impl::InitializationStatus::error_message’ [-Wmissing-field-initializers] 195 return {InitializationStatus::InitializationResult::help_request}; ^ /home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/core/src/impl/Kokkos_Profiling.cpp:201:60: warning: missing initializer for member ‘Kokkos::Tools::Impl::InitializationStatus::error_message’ [-Wmissing-field-initializers] 201 return {InitializationStatus::InitializationResult::success}; ^ [ 26%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_SharedAlloc.cpp.o [ 28%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Spinwait.cpp.o [ 29%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Stacktrace.cpp.o [ 31%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_hwloc.cpp.o [ 33%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/Cuda/Kokkos_CudaSpace.cpp.o [ 35%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/Cuda/Kokkos_Cuda_Instance.cpp.o [ 36%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/Cuda/Kokkos_Cuda_Locks.cpp.o [ 38%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/Cuda/Kokkos_Cuda_Task.cpp.o [ 40%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Instance.cpp.o [ 42%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/OpenMP/Kokkos_OpenMP_Task.cpp.o [ 43%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir///tpls/desul/src/Lock_Array_CUDA.cpp.o [ 45%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir///tpls/desul/src/Lock_Array_HIP.cpp.o [ 47%] Linking CXX static library libkokkoscore.a [ 47%] Built target kokkoscore [ 49%] Building CXX object src/thirdparty/kokkos/containers/src/CMakeFiles/kokkoscontainers.dir/impl/Kokkos_UnorderedMap_impl.cpp.o [ 50%] Linking CXX static library libkokkoscontainers.a [ 50%] Built target kokkoscontainers [ 52%] Building CXX object CMakeFiles/kedm.dir/src/ccm.cpp.o /home/david.wyrick/Git/kEDM/src/ccm.cpp: In function ‘std::vector edm::ccm(edm::TimeSeries, edm::TimeSeries, const std::vector&, int, int, int, int, int, float)’: /home/david.wyrick/Git/kEDM/src/ccm.cpp:254:146: warning: comparison of integer expressions of different signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’ [-Wsign-compare] 254 bool use_scratch = ~~~ ^ [ 54%] Building CXX object CMakeFiles/kedm.dir/src/edim.cpp.o [ 56%] Building CXX object CMakeFiles/kedm.dir/src/io.cpp.o [ 57%] Building CXX object CMakeFiles/kedm.dir/src/simplex.cpp.o [ 59%] Building CXX object CMakeFiles/kedm.dir/src/xmap.cpp.o /home/david.wyrick/Git/kEDM/src/xmap.cpp:46: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas] 46 for (int e = 0; e < E + 1; e++) {

[ 61%] Building CXX object CMakeFiles/kedm.dir/src/knn.cpp.o [ 63%] Building CXX object CMakeFiles/kedm.dir/src/smap.cpp.o [ 64%] Building CXX object CMakeFiles/kedm.dir/src/stats.cpp.o [ 66%] Linking CXX static library libkedm.a [ 66%] Built target kedm [ 68%] Building CXX object CMakeFiles/edm-xmap.dir/src/xmap_run.cpp.o [ 70%] Linking CXX executable edm-xmap [ 70%] Built target edm-xmap [ 71%] Building CXX object CMakeFiles/knn-bench.dir/src/knn_bench.cpp.o [ 73%] Linking CXX executable knn-bench [ 73%] Built target knn-bench [ 75%] Building CXX object CMakeFiles/lookup-bench.dir/src/lookup_bench.cpp.o [ 77%] Linking CXX executable lookup-bench [ 77%] Built target lookup-bench [ 78%] Building CXX object CMakeFiles/smap-bench.dir/src/smap_bench.cpp.o [ 80%] Linking CXX executable smap-bench [ 80%] Built target smap-bench [ 82%] Building CXX object CMakeFiles/_kedm.dir/src/bindings.cpp.o [ 84%] Linking CXX shared module _kedm.cpython-38-x86_64-linux-gnu.so [ 84%] Built target _kedm [ 85%] Building CXX object CMakeFiles/kedm-test.dir/test/main.cpp.o [ 87%] Building CXX object CMakeFiles/kedm-test.dir/test/ccm_test.cpp.o [ 89%] Building CXX object CMakeFiles/kedm-test.dir/test/knn_test.cpp.o [ 91%] Building CXX object CMakeFiles/kedm-test.dir/test/simplex_test.cpp.o [ 92%] Building CXX object CMakeFiles/kedm-test.dir/test/smap_test.cpp.o [ 94%] Building CXX object CMakeFiles/kedm-test.dir/test/stats_test.cpp.o [ 96%] Building CXX object CMakeFiles/kedm-test.dir/test/xmap_one_to_one_test.cpp.o [ 98%] Building CXX object CMakeFiles/kedm-test.dir/test/xmap_all_to_all_test.cpp.o [100%] Linking CXX executable kedm-test Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads For best performance with OpenMP 3.1 set OMP_PROC_BIND=true For unit testing set OMP_PROC_BIND=false

Kokkos::Cuda::initialize WARNING: running kernels compiled for compute capability 8.6 on device with compute capability 8.9 , this will likely reduce potential performance. [100%] Built target kedm-test (kedm2) david.wyrick@robinson:~/Git/kEDM$ ctest build/ Test project /home/david.wyrick/Git/kEDM No tests were found!!! (kedm2) david.wyrick@robinson:~/Git/kEDM$ cd build/ (kedm2) david.wyrick@robinson:~/Git/kEDM/build$ ctest Test project /home/david.wyrick/Git/kEDM/build Start 1: Compute Convergent Cross Mapping 1/19 Test #1: Compute Convergent Cross Mapping ......................... Passed 0.22 sec Start 2: Partially sort kNN LUT 2/19 Test #2: Partially sort kNN LUT ................................... Passed 0.12 sec Start 3: Compute kNN table for E=2 3/19 Test #3: Compute kNN table for E=2 ................................ Passed 0.11 sec Start 4: Compute kNN table for E=3 4/19 Test #4: Compute kNN table for E=3 ................................ Passed 0.13 sec Start 5: Compute kNN table for E=4 5/19 Test #5: Compute kNN table for E=4 ................................ Passed 0.12 sec Start 6: Compute kNN table for E=5 6/19 Test #6: Compute kNN table for E=5 ................................ Passed 0.13 sec Start 7: Compute simplex projection for E=2 7/19 Test #7: Compute simplex projection for E=2 ....................... Passed 0.10 sec Start 8: Compute simplex projection for E=3 8/19 Test #8: Compute simplex projection for E=3 ....................... Passed 0.12 sec Start 9: Compute simplex projection for E=4 9/19 Test #9: Compute simplex projection for E=4 ....................... Passed 0.11 sec Start 10: Compute simplex projection for E=5 10/19 Test #10: Compute simplex projection for E=5 ....................... Passed 0.14 sec Start 11: Compute multivariate simplex projection for E=3 11/19 Test #11: Compute multivariate simplex projection for E=3 .......... Passed 0.11 sec Start 12: Compute optimal embedding dimension 12/19 Test #12: Compute optimal embedding dimension ...................... Passed 0.12 sec Start 13: Compute S-Map projections 13/19 Test #13: Compute S-Map projections ................................Subprocess aborted***Exception: 0.67 sec Start 14: Compute Pearson's correlation coefficient, MAE and MSE 14/19 Test #14: Compute Pearson's correlation coefficient, MAE and MSE ... Passed 0.13 sec Start 15: Compute all-to-all cross mappings 15/19 Test #15: Compute all-to-all cross mappings ........................ Passed 0.14 sec Start 16: Compute one-to-one cross mapping for E=2 16/19 Test #16: Compute one-to-one cross mapping for E=2 ................. Passed 0.11 sec Start 17: Compute one-to-one cross mapping for E=3 17/19 Test #17: Compute one-to-one cross mapping for E=3 ................. Passed 0.11 sec Start 18: Compute one-to-one cross mapping for E=4 18/19 Test #18: Compute one-to-one cross mapping for E=4 ................. Passed 0.12 sec Start 19: Compute one-to-one cross mapping for E=5 19/19 Test #19: Compute one-to-one cross mapping for E=5 ................. Passed 0.10 sec

95% tests passed, 1 tests failed out of 19

Total Test time (real) = 2.92 sec

The following tests FAILED: 13 - Compute S-Map projections (Subprocess aborted) Errors while running CTest Output from these tests are in: /home/david.wyrick/Git/kEDM/build/Testing/Temporary/LastTest.log Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. (kedm2) david.wyrick@robinson:~/Git/kEDM/build$ conda install pytest Collecting package metadata (current_repodata.json): done Solving environment: done

Package Plan

environment location: /home/david.wyrick/anaconda3/envs/kedm2

added / updated specs:

The following NEW packages will be INSTALLED:

exceptiongroup pkgs/main/linux-64::exceptiongroup-1.0.4-py38h06a4308_0 iniconfig pkgs/main/noarch::iniconfig-1.1.1-pyhd3eb1b0_0 pluggy pkgs/main/linux-64::pluggy-1.0.0-py38h06a4308_1 pytest pkgs/main/linux-64::pytest-7.4.0-py38h06a4308_0

Proceed ([y]/n)? y

Downloading and Extracting Packages

Preparing transaction: done Verifying transaction: done Executing transaction: done (kedm2) david.wyrick@robinson:~/Git/kEDM/build$ cd .. (kedm2) david.wyrick@robinson:~/Git/kEDM$ pytest python/tests/ ======================================================= test session starts ======================================================== platform linux -- Python 3.8.17, pytest-7.4.0, pluggy-1.0.0 rootdir: /home/david.wyrick/Git/kEDM plugins: anyio-3.5.0 collected 0 items / 4 errors

============================================================== ERRORS ============================================================== ____ ERROR collecting python/tests/testccm.py ____ ImportError while importing test module '/home/david.wyrick/Git/kEDM/python/tests/test_ccm.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../anaconda3/envs/kedm2/lib/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) python/tests/test_ccm.py:1: in import kedm E ModuleNotFoundError: No module named 'kedm' __ ERROR collecting python/tests/test_simplex.py _____ ImportError while importing test module '/home/david.wyrick/Git/kEDM/python/tests/test_simplex.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../anaconda3/envs/kedm2/lib/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) python/tests/test_simplex.py:1: in import kedm E ModuleNotFoundError: No module named 'kedm' ____ ERROR collecting python/tests/test_smap.py __ ImportError while importing test module '/home/david.wyrick/Git/kEDM/python/tests/test_smap.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../anaconda3/envs/kedm2/lib/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) python/tests/test_smap.py:1: in import kedm E ModuleNotFoundError: No module named 'kedm' ____ ERROR collecting python/tests/test_xmap.py ____ ImportError while importing test module '/home/david.wyrick/Git/kEDM/python/tests/test_xmap.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../anaconda3/envs/kedm2/lib/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) python/tests/test_xmap.py:1: in import kedm E ModuleNotFoundError: No module named 'kedm' ===================================================== short test summary info ====================================================== ERROR python/tests/test_ccm.py ERROR python/tests/test_simplex.py ERROR python/tests/test_smap.py ERROR python/tests/test_xmap.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================== 4 errors in 0.04s ========================================================= (kedm2) david.wyrick@robinson:~/Git/kEDM$ ipython Python 3.8.17 (default, Jul 5 2023, 21:04:15) Type 'copyright', 'credits' or 'license' for more information IPython 8.12.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import kedm

ModuleNotFoundError Traceback (most recent call last) Cell In[1], line 1 ----> 1 import kedm

ModuleNotFoundError: No module named 'kedm'`

keichi commented 1 year ago

Hi, -DKokkos_ARCH_ADA89=ON was unsupported by Kokkos 4.0.00 which was bundled with kEDM. I've updated the Kokkos version in #41.

I think there is still issues with some of the ctests and python tests.

Thanks for finding this out. There was a minor issue in the Smap test code. It has been fixed in #42.

As for the Python bindings, the extension module needs to be manually installed if you build via CMake. An easier way is to use pip to build kEDM:

export CMAKE_ARGS="-DKEDM_ENABLE_GPU=ON -DKokkos_ARCH_ADA89=ON"
pip3 install .
davidgwyrick commented 1 year ago

If I use pip to build kEDM, will it still have GPU support? So I think there are a few things left that need to be updated to support ADA89 architecture.

Running the command: $ cmake -B build -S . -DKEDM_ENABLE_PYTHON=ON -DKEDM_ENABLE_GPU=ON -DKokkos_ARCH_ADA89=ON

completes, but gives me this output / warning:

-- Setting default Kokkos CXX standard to 17 -- Kokkos version: 4.1.00 -- The project name is: Kokkos -- Using internal gtest for testing -- Compiler Version: 11.7.64 -- kokkos_launch_compiler (/home/david.wyrick/Git/kEDM/src/thirdparty/kokkos/bin/kokkos_launch_compiler) is enabled... -- Using -std=c++17 for C++17 standard as feature -- Built-in Execution Spaces: -- Device Parallel: Kokkos::Cuda -- Host Parallel: Kokkos::OpenMP -- Host Serial: NONE

-- Architectures: -- ADA89 -- Using internal desul_atomics copy -- Kokkos Devices: OPENMP;CUDA, Kokkos Backends: OPENMP;CUDA -- Boost.Math: standalone mode ON -- pybind11 v2.9.2 -- kEDM build configuration: -- C++ Compiler: GNU 11.4.0 -- /usr/bin/c++ -- CPU backend: ON -- GPU backend: ON -- Executables: OFF -- Python bindings: ON -- MPI: OFF -- Unit tests: ON -- LIKWID: OFF -- Scratch memory: ON -- SIMD primitives: ON -- Configuring done CMake Warning at src/thirdparty/pybind11/tools/pybind11Tools.cmake:165 (add_library): Cannot generate a safe runtime search path for target _kedm because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm/lib

Some of these libraries may not be found correctly. Call Stack (most recent call first): CMakeLists.txt:136 (pybind11_add_module)

CMake Warning at CMakeLists.txt:151 (add_executable): Cannot generate a safe runtime search path for target kedm-test because files in some directories may conflict with libraries in implicit directories:

runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-linux-gnu/11 may be hidden by files in:
  /home/david.wyrick/anaconda3/envs/kedm/lib

Some of these libraries may not be found correctly.

-- Generating done -- Build files have been written to: /home/david.wyrick/Git/kEDM/build

Attempting to complete the build process: $ cmake --build build [ 2%] Building CXX object src/thirdparty/kokkos/simd/src/CMakeFiles/kokkossimd.dir/Kokkos_SIMD_dummy.cpp.o [ 4%] Linking CXX static library libkokkossimd.a [ 4%] Built target kokkossimd [ 6%] Building CXX object src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o nvcc fatal : Value 'sm_89' is not defined for option 'gpu-architecture' gmake[2]: [src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/build.make:76: src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_CPUDiscovery.cpp.o] Error 1 gmake[1]: [CMakeFiles/Makefile2:1211: src/thirdparty/kokkos/core/src/CMakeFiles/kokkoscore.dir/all] Error 2 gmake: *** [Makefile:146: all] Error 2

pip3 install . gives similar errors about "sm_89" not being defined for option "gpu-architecture"

keichi commented 1 year ago

Yes, it will have GPU support if you build from source using pip. If you install a prebuilt wheel from PyPi (pip install kedm) it won't have GPU support.

Can you confirm if the nvcc in your PATH supports Ada-gen GPUs (sm_89)? Your nvcc might be pointing to an older CUDA installation. I was able to compile kEDM for ADA89 on my workstation.

keichi@dango:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
keichi@dango:~$ nvcc --list-gpu-code
sm_50
sm_52
sm_53
sm_60
sm_61
sm_62
sm_70
sm_72
sm_75
sm_80
sm_86
sm_87
sm_89
sm_90
davidgwyrick commented 1 year ago

ahh I'm using cuda toolkit 11.7, which only supports up to sm_87. I did that because kokkos is only tested to 11.7.

12.1 seems to work. The ctest and python tests all passed now and it looks like I can run the python bindings on data. Although kedm.xmap doesn't look like it's using the GPU to compute the cross embedding. Is that something that only works from the executable?

and while I have you, what does the "Tp – Prediction interval" parameter in kedm.xmap represent? The documentation says it's an integer, between 0 & 100? Is it the ratio of data used for testing vs prediction?

davidgwyrick commented 1 year ago

@keichi if you have a moment to answer the questions in the previous comment, I would really appreciate it!

1) Is GPU compatibility only available for the executable or should it also work for the python bindings? 2) what does the "Tp – Prediction interval" parameter in kedm.xmap represent? The documentation says it's an integer, between 0 & 100? Is it the ratio of data used for testing vs prediction? 3) Is there an option for the cross-mapping functions to randomly project the delay coordinate vectors ?

keichi commented 1 year ago

Apologies for my delayed response.

Is GPU compatibility only available for the executable or should it also work for the python bindings?

The executables should also use the GPU if kEDM was built with GPU enabled. How did you test if the GPU is used?

what does the "Tp – Prediction interval" parameter in kedm.xmap represent? The documentation says it's an integer, between 0 & 100? Is it the ratio of data used for testing vs prediction?

Tp is how many time steps you predict in the future. xmap predicts the target time series at time step t+Tp using the library time series at time step t (and its delays). It then evaluates the prediction skill by comparing the prediction and the actual target values.

Is there an option for the cross-mapping functions to randomly project the delay coordinate vectors?

Unfortunately, no.

davidgwyrick commented 1 year ago

The executables should also use the GPU if kEDM was built with GPU enabled. How did you test if the GPU is used?

When I run ctest in the build directory, the tests finish in like 8seconds and appear to use the GPU. The pytests take 6minutes on the other hand. Do you have a better way of testing GPU functionality?

xmap predicts the target time series at time step t+Tp using the library time series at time step t (and its delays).

The standard is looking at just 1 timestep ahead in the future correct? Like all of the papers that have come out thus far.

Unfortunately, no.

Is this a feature that could be added easily? I think for our time series data this is important.

Thank you! Cheers from Seattle, USA

keichi commented 1 year ago

The C++ and Python unit tests are meant to verify the functionality and correctness of the library; they should not be used for performance measurements. One way to monitor GPU utilization is the nvidia-smi utility that comes with CUDA. If you run nvidia-smi pmon it will continuously print out the GPU utilization and processes that use the GPU. If kEDM shows up there it is using the GPU.

Actually, I think Tp=0 is more common in CCM. The default Tp in Sugihara Lab's reference EDM implementation is also zero. https://sugiharalab.github.io/EDM_Documentation/edm_functions/#ccm For Simplex, yes Tp=1 is standard.

Is this a feature that could be added easily? I think for our time series data this is important.

I need to first understand what exactly you need... Do you have a paper or something that describes the algorithm?

davidgwyrick commented 1 year ago

I need to first understand what exactly you need... Do you have a paper or something that describes the algorithm?

https://doi.org/10.1371/journal.pcbi.1004537

In the methods section 'Random coordinate cross-embedding', they explain that projecting the delay coordinate vectors with a random matrix helps with different timescales of each timeseries. Should be pretty simple addition to the code.

davidgwyrick commented 1 year ago

@keichi , I'm trying to understand the difference between the edm-xmap executable and the python binding xmap. Are they different? In the documentation, it says the executable performs pairwise cross mapping using simplex projection. What does the python binding xmap do? Furthermore, how are the xmap and ccm python functions different? Are the random samples drawn in the CCM function excluded from the embedding procedure?

A follow up question is why the python bindings run slower than the executable. Both appear to be using the GPU, but the executable calculates the all-to-all CCM for a 10,000 x 500 matrix in like 20 seconds whereas the python code takes forever (i didn't finish it). it's almost as if the python binding is only partially utilizing the GPU.

Apologize for the block of questions. If you think it's better to chat in a different place than your git issues, we can continue elsewhere. Thank you.

keichi commented 1 year ago

You are welcome to ask questions on GitHub issues. I transferred your question to a new issue so it's easier for other people to search.