search

getkeops / keops

KErnel OPerationS, on CPUs and GPUs, with autodiff and without memory overflows
https://www.kernel-operations.io
MIT License
1.04k stars 64 forks source link

FileNotFoundError: [Errno 2] No such file or directory: '/.cache/pykeops-1.5-cpython-36//KeOps_formula.o #170

Open Nandana21 opened 3 years ago

Nandana21 commented 3 years ago

Hi All, I'm trying to get pykeops version 1.5 running. we have CUDA 10.1.240 and gcc, g++ 7.5. However, when I try to install pykeops and run a sample test code on my gpu notebook, i run into the following error

Can you please help here?

!pip install cmake
!pip install pykeops

import numpy as np
import pykeops
pykeops.clean_pykeops()          # just in case old build files are still present
print(pykeops.config.bin_folder)

pykeops.config.verbose = True
from pykeops.numpy import Genred

x = np.arange(1, 10).reshape(-1, 3).astype('float32')
y = np.arange(3, 9 ).reshape(-1, 3).astype('float32')

my_conv = Genred('-SqNorm2(x-y)', ['x = Vi(3)', 'y = Vj(3)'])
print(my_conv(x, y))

image image image

hypnopump commented 3 years ago

Same issue here. Any workaround/solution?

Closing #171 because i Installed the cudatoolkit-dev via anaconda ( https://anaconda.org/conda-forge/cudatoolkit-dev ) to get nvcc and now have this same error.

Trace: 

In [1]: import pykeops
   ...: pykeops.clean_pykeops()          # just in case old build files are still present
   ...: pykeops.test_numpy_bindings()    # perform the compilation
Cleaning /home/hypnopump/.cache/pykeops-1.5-cpython-37/...
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_d003191010 has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450 has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-133d87d50e has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_7d11209174 has been removed.
[pyKeOps] Initializing build folder for dtype=float64 and lang=numpy in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... CMake Error at /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/cuda.cmake:107 (cmake_minimum_required):
  CMake 3.18 or higher is required.  You are running version 3.16.3
Call Stack (most recent call first):
  CMakeLists.txt:29 (include)

--------------------- CMAKE DEBUG -----------------
Command '['cmake', '/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/', "-DCMAKE_BUILD_TYPE='Release'", "-Dshared_obj_name='KeOps_formula'", "-DPYTHON_LANG='numpy'", "-D__TYPE__='double'", '-DC_CONTIGUOUS=1', "-DcommandLine=cmake /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/ -DCMAKE_BUILD_TYPE='Release' -Dshared_obj_name='KeOps_formula' -DPYTHON_LANG='numpy' -D__TYPE__='double' -DC_CONTIGUOUS=1"]' returned non-zero exit status 1.
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /home/hypnopump/miniconda3/envs/assistant/bin/nvcc
-- The CUDA Host CXX Compiler: /usr/bin/c++
-- The CUDA compiler identification is NVIDIA 11.0.221
-- Check for working CUDA compiler: /home/hypnopump/miniconda3/envs/assistant/bin/nvcc
-- Check for working CUDA compiler: /home/hypnopump/miniconda3/envs/assistant/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Compute properties automatically set to: -DMAXIDGPU=3;-DMAXTHREADSPERBLOCK0=1024;-DSHAREDMEMPERBLOCK0=49152;-DMAXTHREADSPERBLOCK1=1024;-DSHAREDMEMPERBLOCK1=49152;-DMAXTHREADSPERBLOCK2=1024;-DSHAREDMEMPERBLOCK2=49152;-DMAXTHREADSPERBLOCK3=1024;-DSHAREDMEMPERBLOCK3=49152
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Autodetected CUDA architecture(s):  8.6 8.6 8.6 8.6
-- Configuring incomplete, errors occurred!
See also "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeOutput.log".
See also "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeError.log".

--------------------- ----------- -----------------
done.
[pyKeOps] Compiling libKeOpsnumpy49510d3d8d in /home/hypnopump/.cache/pykeops-1.5-cpython-37:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3); 
       dtype  : float64
... 
make: *** No rule to make target 'KeOps_formula'.  Stop.

--------------------- MAKE DEBUG -----------------
Command '['cmake', '--build', '.', '--target', 'KeOps_formula', '--', 'VERBOSE=1']' returned non-zero exit status 2.

--------------------- ----------- -----------------
[pyKeOps] Compiling pybind11 template libKeOps_template_7d11209174 in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... [pyKeOps] Initializing build folder for dtype=float64 and lang=numpy in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... CMake Error at /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/cuda.cmake:107 (cmake_minimum_required):
  CMake 3.18 or higher is required.  You are running version 3.16.3
Call Stack (most recent call first):
  CMakeLists.txt:29 (include)

--------------------- CMAKE DEBUG -----------------
Command '['cmake', '/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/', "-DCMAKE_BUILD_TYPE='Debug'", "-Dshared_obj_name='KeOps_formula'", "-DPYTHON_LANG='numpy'", "-D__TYPE__='double'", '-DC_CONTIGUOUS=1', "-DcommandLine=cmake /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/ -DCMAKE_BUILD_TYPE='Debug' -Dshared_obj_name='KeOps_formula' -DPYTHON_LANG='numpy' -D__TYPE__='double' -DC_CONTIGUOUS=1"]' returned non-zero exit status 1.
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /home/hypnopump/miniconda3/envs/assistant/bin/nvcc
-- The CUDA Host CXX Compiler: /usr/bin/c++
-- The CUDA compiler identification is NVIDIA 11.0.221
-- Check for working CUDA compiler: /home/hypnopump/miniconda3/envs/assistant/bin/nvcc
-- Check for working CUDA compiler: /home/hypnopump/miniconda3/envs/assistant/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Compute properties automatically set to: -DMAXIDGPU=3;-DMAXTHREADSPERBLOCK0=1024;-DSHAREDMEMPERBLOCK0=49152;-DMAXTHREADSPERBLOCK1=1024;-DSHAREDMEMPERBLOCK1=49152;-DMAXTHREADSPERBLOCK2=1024;-DSHAREDMEMPERBLOCK2=49152;-DMAXTHREADSPERBLOCK3=1024;-DSHAREDMEMPERBLOCK3=49152
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Autodetected CUDA architecture(s):  8.6 8.6 8.6 8.6
-- Configuring incomplete, errors occurred!
See also "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-133d87d50e/CMakeFiles/CMakeOutput.log".
See also "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-133d87d50e/CMakeFiles/CMakeError.log".

--------------------- ----------- -----------------
done.
[pyKeOps] Compiling libKeOpsnumpy49510d3d8d in /home/hypnopump/.cache/pykeops-1.5-cpython-37:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3); 
       dtype  : float64
... 
make: *** No rule to make target 'KeOps_formula'.  Stop.

--------------------- MAKE DEBUG -----------------
Command '['cmake', '--build', '.', '--target', 'KeOps_formula', '--', 'VERBOSE=1']' returned non-zero exit status 2.

--------------------- ----------- -----------------
[pyKeOps] Compiling pybind11 template libKeOps_template_d003191010 in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... ---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/test/install.py in test_numpy_bindings()
     21 
---> 22         my_conv = pknp.Genred(formula, var)
     23         if np.allclose(my_conv(x, y).flatten(), expected_res):

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.py in __init__(self, formula, aliases, reduction_op, axis, dtype, opt_arg, formula2, cuda_type, dtype_acc, use_double_acc, sum_scheme, enable_chunks, optional_flags, rec_multVar_highdim)
    182         self.myconv = LoadKeOps(
--> 183             self.formula, self.aliases, self.dtype, "numpy", self.optional_flags
    184         ).import_module()

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/common/keops_io.py in __init__(self, formula, aliases, dtype, lang, optional_flags, include_dirs)
     47         ):
---> 48             self._safe_compile()
     49 

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/common/utils.py in wrapper_filelock(*args, **kwargs)
     74                 with FileLock(f):
---> 75                     func_res = func(*args, **kwargs)
     76 

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/common/keops_io.py in _safe_compile(self)
     62             self.include_dirs,
---> 63             self.build_folder,
     64         )

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/common/compile_routines.py in compile_generic_routine(formula, aliases, dllname, dtype, lang, optional_flags, include_dirs, build_folder)
    245     template_name, is_rebuilt = get_or_build_pybind11_template(
--> 246         dtype, lang, include_dirs, use_prebuilt_formula=True
    247     )

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/common/compile_routines.py in get_or_build_pybind11_template(dtype, lang, include_dirs, use_prebuilt_formula)
     87             pykeops.config.bin_folder + os.path.sep + fname,
---> 88             template_build_folder + os.path.sep + fname,
     89         )

FileNotFoundError: [Errno 2] No such file or directory: '/home/hypnopump/.cache/pykeops-1.5-cpython-37//KeOps_formula.o' -> '/home/hypnopump/.cache/pykeops-1.5-cpython-37//build-pybind11_template-libKeOps_template_7d11209174/KeOps_formula.o'

I inspected the CMake log files as suggested. Here's the /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeError.log: 

Performing C++ SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_73365/fast && /usr/bin/make -f CMakeFiles/cmTC_73365.dir/build.make CMakeFiles/cmTC_73365.dir/build
make[1]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_73365.dir/src.cxx.o
/usr/bin/c++    -DCMAKE_HAVE_LIBC_PTHREAD   -std=gnu++14 -o CMakeFiles/cmTC_73365.dir/src.cxx.o -c /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp/src.cxx
Linking CXX executable cmTC_73365
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_73365.dir/link.txt --verbose=1
/usr/bin/c++   -DCMAKE_HAVE_LIBC_PTHREAD    CMakeFiles/cmTC_73365.dir/src.cxx.o  -o cmTC_73365 
/usr/bin/ld: CMakeFiles/cmTC_73365.dir/src.cxx.o: in function `main':
src.cxx:(.text+0x46): undefined reference to `pthread_create'
/usr/bin/ld: src.cxx:(.text+0x52): undefined reference to `pthread_detach'
/usr/bin/ld: src.cxx:(.text+0x63): undefined reference to `pthread_join'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_73365.dir/build.make:87: cmTC_73365] Error 1
make[1]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_73365/fast] Error 2

Source file was:
#include <pthread.h>

void* test_func(void* data)
{
  return data;
}

int main(void)
{
  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_4b67f/fast && /usr/bin/make -f CMakeFiles/cmTC_4b67f.dir/build.make CMakeFiles/cmTC_4b67f.dir/build
make[1]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_4b67f.dir/CheckFunctionExists.cxx.o
/usr/bin/c++    -DCHECK_FUNCTION_EXISTS=pthread_create   -std=gnu++14 -o CMakeFiles/cmTC_4b67f.dir/CheckFunctionExists.cxx.o -c /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CheckLibraryExists/CheckFunctionExists.cxx
Linking CXX executable cmTC_4b67f
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_4b67f.dir/link.txt --verbose=1
/usr/bin/c++   -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_4b67f.dir/CheckFunctionExists.cxx.o  -o cmTC_4b67f  -lpthreads 
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_4b67f.dir/build.make:87: cmTC_4b67f] Error 1
make[1]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-f0b4802450/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_4b67f/fast] Error 2
hypnopump commented 3 years ago

Hi there! Tried installing in a machine with no CUDA GPUs, and it worked in CPU. However, when installing in a machine with CUDA, it always tries to use the available GPUs and won't even work in the CPU as a fallback.

Is there any way of hiding the available GPUs/CUDA devices from pykeops?

joanglaunes commented 3 years ago

Hello @hypnopump , About your last question, you can hide GPU devices by setting the environment variable CUDA_VISIBLE_DEVICES to be empty, as follows: export CUDA_VISIBLE_DEVICES= Now, about your previous problem and the one mentioned by @Nandana21 , there are clearly many possible issues with the interplay between the configuration of cmake, nvcc, gcc etc. One thing is the Cmake version which needs to be 3.18 or higher, but I do not think that this is the problem in both cases (but just to make sure, please check your cmake version and update for example via pip or anaconda if lower than 3.18). I see in the error log you posted that there is an error with "pthreads not found". I know we already discussed similar issues previously ; possibly this "pthreads" issue is misleading and hiding another issue, as was discussed for issue #142 . I hope maybe @bcharlier or @jeanfeydy can have some ideas to fix all this. Now, we are working on a new version that should not use cmake, and hopefully it will solve all these problems in a few weeks...

hypnopump commented 3 years ago

SOLVED! Mine was indeed a CMake error (i had 3.16). I got it working... thanks! Let's see if @Nandana21 's problem is the same...

hypnopump commented 3 years ago

Unfortunately, it seem seems it only solved the cpu case... the GPU keeps failing.

Got this message: 

In [1]: import pykeops
   ...: pykeops.clean_pykeops()          # just in case old build files are still present
   ...: pykeops.test_numpy_bindings()    # perform the compilation
Cleaning /home/hypnopump/.cache/pykeops-1.5-cpython-37/...
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/keops_hash.log has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_0b22393cf1 has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-1e7eaa8344 has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/libKeOpsnumpy49510d3d8d has been removed.
    - /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f has been removed.
[pyKeOps] Initializing build folder for dtype=float64 and lang=numpy in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... done.
[pyKeOps] Compiling libKeOpsnumpy49510d3d8d in /home/hypnopump/.cache/pykeops-1.5-cpython-37:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3); 
       dtype  : float64
... 
[pyKeOps] Compiling pybind11 template libKeOps_template_0b22393cf1 in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... done.
Done.
[pyKeOps] Initializing build folder for dtype=float64 and lang=numpy in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... -- The CXX compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- No CUDA support
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f

done.
[pyKeOps] Compiling libKeOpsnumpy49510d3d8d in /home/hypnopump/.cache/pykeops-1.5-cpython-37:
       formula: Sum_Reduction(SqNorm2(x - y),1)
       aliases: x = Vi(0,3); y = Vj(1,3); 
       dtype  : float64
... 
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -S/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula -B/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/make  -f CMakeFiles/Makefile2 KeOps_formula
make[1]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -S/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula -B/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f --check-build-system CMakeFiles/Makefile.cmake 0
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_progress_start /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles 1
/usr/bin/make  -f CMakeFiles/Makefile2 CMakeFiles/KeOps_formula.dir/all
make[2]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/usr/bin/make  -f CMakeFiles/copy_KeOps_formula.dir/build.make CMakeFiles/copy_KeOps_formula.dir/depend
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
cd /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f && /home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_depends "Unix Makefiles" /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/copy_KeOps_formula.dir/DependInfo.cmake --color=
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/copy_KeOps_formula.dir/DependInfo.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/copy_KeOps_formula.dir/depend.internal".
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/copy_KeOps_formula.dir/depend.internal".
Scanning dependencies of target copy_KeOps_formula
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/usr/bin/make  -f CMakeFiles/copy_KeOps_formula.dir/build.make CMakeFiles/copy_KeOps_formula.dir/build
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
[100%] Building CXX object CMakeFiles/copy_KeOps_formula.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/core/link_autodiff.cpp.o
/home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ -DC_CONTIGUOUS=1 -DUSE_CUDA=0 -DUSE_DOUBLE=1 -DUSE_HALF=0 -D__TYPE__=double -I/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/../.. -I/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f -I/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula/../../keops -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/hypnopump/miniconda3/envs/assistant/include -DUSE_OPENMP -fopenmp -Wall -Wno-unknown-pragmas -fmax-errors=2 -g -O0 -g -fPIC -include KeOps_formula.h -std=gnu++14 -o CMakeFiles/copy_KeOps_formula.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/core/link_autodiff.cpp.o -c /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/core/link_autodiff.cpp
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
[100%] Built target copy_KeOps_formula
/usr/bin/make  -f CMakeFiles/KeOps_formula.dir/build.make CMakeFiles/KeOps_formula.dir/depend
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
cd /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f && /home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_depends "Unix Makefiles" /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_keops_formula /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/KeOps_formula.dir/DependInfo.cmake --color=
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/KeOps_formula.dir/DependInfo.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/KeOps_formula.dir/depend.internal".
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/KeOps_formula.dir/depend.internal".
Scanning dependencies of target KeOps_formula
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/usr/bin/make  -f CMakeFiles/KeOps_formula.dir/build.make CMakeFiles/KeOps_formula.dir/build
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E copy /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles/copy_KeOps_formula.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/keops/core/link_autodiff.cpp.o /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/..//KeOps_formula.o
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
[100%] Built target KeOps_formula
make[2]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_progress_start /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f/CMakeFiles 0
make[1]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-4ee0138b7f'

[pyKeOps] Compiling pybind11 template libKeOps_template_55aac40c77 in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... -- The CXX compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- No CUDA support
-- pybind11 v2.6.1 
-- Found PythonInterp: /home/hypnopump/miniconda3/envs/assistant/bin/python (found suitable version "3.7.10", minimum required is "3.7") 
-- Found PythonLibs: /home/hypnopump/miniconda3/envs/assistant/lib/libpython3.7m.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77

/home/hypnopump/miniconda3/envs/assistant/bin/cmake -S/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template -B/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/make  -f CMakeFiles/Makefile2 libKeOps_template_55aac40c77
make[1]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -S/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template -B/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 --check-build-system CMakeFiles/Makefile.cmake 0
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_progress_start /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles 2
/usr/bin/make  -f CMakeFiles/Makefile2 CMakeFiles/libKeOps_template_55aac40c77.dir/all
make[2]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
/usr/bin/make  -f CMakeFiles/libKeOps_template_55aac40c77.dir/build.make CMakeFiles/libKeOps_template_55aac40c77.dir/depend
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
cd /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 && /home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_depends "Unix Makefiles" /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles/libKeOps_template_55aac40c77.dir/DependInfo.cmake --color=
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles/libKeOps_template_55aac40c77.dir/DependInfo.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles/libKeOps_template_55aac40c77.dir/depend.internal".
Dependee "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles/libKeOps_template_55aac40c77.dir/depend.internal".
Scanning dependencies of target libKeOps_template_55aac40c77
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
/usr/bin/make  -f CMakeFiles/libKeOps_template_55aac40c77.dir/build.make CMakeFiles/libKeOps_template_55aac40c77.dir/build
make[3]: Entering directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
[ 50%] Building CXX object CMakeFiles/libKeOps_template_55aac40c77.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.cpp.o
/home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ -DC_CONTIGUOUS=1 -DMODULE_NAME=libKeOps_template_55aac40c77 -DUSE_CUDA=0 -DUSE_DOUBLE=1 -DUSE_HALF=0 -D__TYPE__=double -DlibKeOps_template_55aac40c77_EXPORTS -I/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template/../.. -I/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77 -I/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/cmake_scripts/script_template/../../keops -isystem /home/hypnopump/miniconda3/envs/assistant/include/python3.7m -isystem /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/pybind11/include -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/hypnopump/miniconda3/envs/assistant/include -DUSE_OPENMP -fopenmp -Wall -Wno-unknown-pragmas -fmax-errors=2 -g -O0 -g -fPIC -fvisibility=hidden -std=gnu++14 -o CMakeFiles/libKeOps_template_55aac40c77.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.cpp.o -c /home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.cpp
[100%] Linking CXX shared module libKeOps_template_55aac40c77.cpython-37m-x86_64-linux-gnu.so
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_link_script CMakeFiles/libKeOps_template_55aac40c77.dir/link.txt --verbose=1
/home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ -fPIC -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/hypnopump/miniconda3/envs/assistant/include -DUSE_OPENMP -fopenmp -Wall -Wno-unknown-pragmas -fmax-errors=2 -g -O0 -g -fopenmp -Wl,-rpath,$ORIGIN -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/hypnopump/miniconda3/envs/assistant/lib -Wl,-rpath-link,/home/hypnopump/miniconda3/envs/assistant/lib -L/home/hypnopump/miniconda3/envs/assistant/lib -shared  -o libKeOps_template_55aac40c77.cpython-37m-x86_64-linux-gnu.so CMakeFiles/libKeOps_template_55aac40c77.dir/home/hypnopump/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.cpp.o KeOps_formula.o 
make[3]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
[100%] Built target libKeOps_template_55aac40c77
make[2]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'
/home/hypnopump/miniconda3/envs/assistant/bin/cmake -E cmake_progress_start /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77/CMakeFiles 0
make[1]: Leaving directory '/home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77'

done.
Done.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/test/install.py in test_numpy_bindings()
     22         my_conv = pknp.Genred(formula, var)
---> 23         if np.allclose(my_conv(x, y).flatten(), expected_res):
     24             print("\npyKeOps with numpy bindings is working!\n")

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.py in __call__(self, backend, device_id, ranges, *args)
    313         out = self.myconv.genred_numpy(
--> 314             tagCpuGpu, tag1D2D, 0, device_id, ranges, nx, ny, *args
    315         )

RuntimeError: [KeOps] This KeOps shared object has been compiled without cuda support: 
 1) to perform computations on CPU, simply set tagHostDevice to 0
 2) to perform computations on GPU, please recompile the formula with a working version of cuda.

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-1-eb2849f7dd26> in <module>
      1 import pykeops
      2 pykeops.clean_pykeops()          # just in case old build files are still present
----> 3 pykeops.test_numpy_bindings()    # perform the compilation

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/test/install.py in test_numpy_bindings()
     34         pykeops.config.build_type = "Debug"
     35         my_conv = pknp.Genred(formula, var)
---> 36         print(my_conv(x, y))
     37 
     38 

~/miniconda3/envs/assistant/lib/python3.7/site-packages/pykeops/numpy/generic/generic_red.py in __call__(self, backend, device_id, ranges, *args)
    312 
    313         out = self.myconv.genred_numpy(
--> 314             tagCpuGpu, tag1D2D, 0, device_id, ranges, nx, ny, *args
    315         )
    316 

RuntimeError: [KeOps] This KeOps shared object has been compiled without cuda support: 
 1) to perform computations on CPU, simply set tagHostDevice to 0
 2) to perform computations on GPU, please recompile the formula with a working version of cuda.

But nvcc is installed via : https://anaconda.org/trenta3/cudatoolkit-dev

bash
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

And the pytorch version is 1.8 with cudatoolkit being 10.2. Any hints as to what might be the underlying issue?

PD: I'm running all this with conda installations since I don't have sudo permissions in that machine.

hypnopump commented 3 years ago

Okay, kinda solved using @jeanfeydy 's env from https://gist.github.com/jeanfeydy/d9395912e4d8bf1c0340bc93f78ea4bd

joanglaunes commented 3 years ago

Hello @hypnopump , Ok good to hear that it solved the problem. But from your error message above apparently cmake was not finding the Cuda compiler nvcc. This command should be in your path, maybe this was the issue ? So in fact pykeops detected that you can use a Gpu but then when compiling the formula Cmake was not detecting nvcc so compiled the formula for Cpu only.

hypnopump commented 3 years ago

Hi @joanglaunes , I don't knnow exactly which was the problem. nvcc was in the path and calling it worked (see the nvcc --version which returned its version). I suspect it had something to to with the nvcc-cuda version compatibility, although i tried what i thoughtwas a valid convination from the list u guys referenced at some point. Anyway, I got everything working after using a custom modification of @jeanfeydy 's env.

Originofamonia commented 2 years ago

Hi,

Same error. so how to solve this?

patel-zeel commented 2 years ago

Hi,

Same error. so how to solve this?

I suppose the main problem identifier lies in the following lines. In my case, nvcc was corrected and resolving that solved the issue.

[pyKeOps] Compiling pybind11 template libKeOps_template_55aac40c77 in /home/hypnopump/.cache/pykeops-1.5-cpython-37 ... -- The CXX compiler identification is GNU 9.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/hypnopump/miniconda3/envs/assistant/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- No CUDA support
-- pybind11 v2.6.1 
-- Found PythonInterp: /home/hypnopump/miniconda3/envs/assistant/bin/python (found suitable version "3.7.10", minimum required is "3.7") 
-- Found PythonLibs: /home/hypnopump/miniconda3/envs/assistant/lib/libpython3.7m.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/hypnopump/.cache/pykeops-1.5-cpython-37/build-pybind11_template-libKeOps_template_55aac40c77
izzatum commented 2 years ago

Hi @patel-zeel. How do you fix the issue? Do you mind sharing how you corrected the nvcc? Thanks

patel-zeel commented 2 years ago

Hi @izzatum. First, let me clarify that nvcc problem is totally unrelated to KeOps. Now, I followed one of the Cuda drivers+toolkit installation guide (The key was to set PATH and LD_LIBRARY_PATH after installation). You can find more details here.

izzatum commented 2 years ago

Thanks for sharing @patel-zeel! In my case, I tried both on my local machines with and without GPU. Still, I couldn't make it work. I also tried to run it on the server but still failed. I don't know already what to do.

Celia0971 commented 2 years ago

Same problem. FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.cache/pykeops-1.5-cpython-38//KeOps_formula.o' -> '/home/user/.cache/pykeops-1.5-cpython-38//build-pybind11_template-libKeOps_template_5558aff716/KeOps_formula.o'
how to fix this?

jeanfeydy commented 2 years ago

Hi @Celia0971 ,

We are currently in the process of releasing KeOps 2.0, which has an entirely different (and much simple) compilation pipeline. There is a good chance that this new engine will have more luck locating CUDA on your machine. Could you try to install it with:

pip install git+https://github.com/getkeops/keops.git@python_engine

And let us know if the problem is still there?

Best regards, Jean

jeanfeydy commented 2 years ago

By the way: I know that as of today, on Google Colab, installing KeOps with !pip install pykeops will throw this error. As detailed in the doc, please make sure that you install all the required dependencies with:

!pip install pykeops[colab]

instead :-)

Best regards, Jean

leieric commented 2 years ago

Hi @Celia0971 ,

We are currently in the process of releasing KeOps 2.0, which has an entirely different (and much simple) compilation pipeline. There is a good chance that this new engine will have more luck locating CUDA on your machine. Could you try to install it with:

pip install git+https://github.com/getkeops/keops.git@python_engine

And let us know if the problem is still there?

Best regards, Jean

Just wanted to confirm that installing pykeops using this command worked for me. I was having the same installation issues as everyone above, but the new compilation engine seems to have solved it. Thank you!

izzatum commented 2 years ago

Hi @jeanfeydy,

I hope you're doing well. I tried to install using pip install git+https://github.com/getkeops/keops.git@python_engine. Unfortunately, I received errors as follows:

`Collecting git+https://github.com/getkeops/keops.git@python_engine Cloning https://github.com/getkeops/keops.git (to revision python_engine) to /private/var/folders/3q/yps71nyj565b2g31vrgbwr50fztv4w/T/pip-req-build-3z8ib6d0 ERROR: Command errored out with exit status 1: command: git version cwd: None Complete output (2 lines): xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

WARNING: Discarding git+https://github.com/getkeops/keops.git@python_engine. Command errored out with exit status 1: git version Check the logs for full command output. ERROR: Command errored out with exit status 1: git version Check the logs for full command output.`

Could you assist me with that?

Thank you!

joanglaunes commented 2 years ago

Hello @izzatum , This error has nothing to do with KeOps, it is about installing the developer tools on Mac. It think you have to do the command xcode-select --install