search

src-d / kmcuda

Large scale K-means and K-nn implementation on NVIDIA GPU / CUDA
Other
791 stars 145 forks source link

install issue #27

Closed QinglongWang closed 6 years ago

QinglongWang commented 6 years ago

I created a virtual env with conda, with python 3.6. Then I installed as instructed:

pip install libKMCUDA

Gives me following error:

Collecting libKMCUDA Could not find a version that satisfies the requirement libKMCUDA (from versions: 1.0.0.linux-x86_64, 1.0.1.linux-x86_64, 1.0.2.linux-x86_64) No matching distribution found for libKMCUDA

Then I also tried:

pip install git+https://github.com/src-d/kmcuda.git#subdirectory=src

Gives me following error:

Collecting git+https://github.com/src-d/kmcuda.git#subdirectory=src Cloning https://github.com/src-d/kmcuda.git to /tmp/pip-ab5m6_e8-build Requirement already satisfied: numpy in ./py3_env/lib/python3.4/site-packages (from libKMCUDA==6.2.0) Installing collected packages: libKMCUDA Running setup.py install for libKMCUDA ... error Complete output from command /home/qlw/miniconda2/envs/py3_env/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-ab5m6_e8-build/src/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-r1fc33gf-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_py -- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Try OpenMP C flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Success -- Try OpenMP CXX flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Success -- Found OpenMP: -fopenmp CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message): Could NOT find CUDA (missing: CUDA_CUDART_LIBRARY) (found version "9.0") Call Stack (most recent call first): /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE) /usr/share/cmake-3.5/Modules/FindCUDA.cmake:949 (find_package_handle_standard_args) CMakeLists.txt:10 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/pip-ab5m6_e8-build/src/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-ab5m6_e8-build/src/setup.py", line 79, in <module>
    "Programming Language :: Python :: 3.5"
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/command/install.py", line 61, in run
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/command/install.py", line 539, in run
    self.run_command('build')
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/command/build.py", line 126, in run
    self.run_command(cmd_name)
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-ab5m6_e8-build/src/setup.py", line 19, in run
    self._build()
  File "/tmp/pip-ab5m6_e8-build/src/setup.py", line 34, in _build
    "-DCUDA_TOOLKIT_ROOT_DIR=%s" % cuda_toolkit_dir, "."))
  File "/home/qlw/miniconda2/envs/py3_env/lib/python3.4/subprocess.py", line 558, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('cmake', '-DCMAKE_BUILD_TYPE=Release', '-DDISABLE_R=y', '-DCUDA_TOOLKIT_ROOT_DIR=:/usr/local/cuda/bin', '.')' returned non-zero exit status 1

----------------------------------------

Command "/home/qlw/miniconda2/envs/py3_env/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-ab5m6_e8-build/src/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-r1fc33gf-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-ab5m6_e8-build/src

Any help? Thanks

vmarkovtsev commented 6 years ago

Hi! Thanks for reporting.

The first pip install did not work because there are no prebuilt wheels for 3.4 - yours is 3.4, not 3.6 as clearly seen in the log.

The second did not work because the build was not tested with CUDA 9.0. Upgrading your cmake to the latest version may or may not help. Check the value of CUDA_TOOLKIT_ROOT_DIR in your log from above - is it correct? Otherwise it makes sense to export CUDA_TOOLKIT_ROOT_DIR=/whatever/correct

QinglongWang commented 6 years ago

Thanks for the response. Yes I created the virtual env with wrong version of python. I corrected that by specifying python=3.5 when creating the conda env. Then I hit this error which very similar to this one, except that I was suggested to build kmcuda with -DCUDA_ARCH=35. Following is the detail info the error:

reassignments threshold: 100 compute capability mismatch for device 0: wanted 6.1, have 3.5

you may want to build kmcuda with -DCUDA_ARCH=35 (refer to "Building" in README.md) compute capability mismatch for device 1: wanted 6.1, have 3.5 you may want to build kmcuda with -DCUDA_ARCH=35 (refer to "Building" in README.md) Traceback (most recent call last): File "test_kmcuda_1.py", line 11, in centroids, assignments = kmeans_cuda(arr, 4, verbosity=1, seed=3) ValueError: No such CUDA device exists

Like commented in that previous issue, I tried to build manually instead of installing with pip. Here is the cmd I used to build,

(py35_env/) [qxw41@lrs-xing01 src]$ /usr/local/bin/cmake -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=/home/qxw41/anaconda2/envs/py35_env/bin/python3.5m -DPYTHON_LIBRARY=/home/qxw41/anaconda2/envs/py35_env/lib/libpython3.5m.so -DPYTHON_INCLUDE_DIR=/home/qxw41/anaconda2/envs/py35_env/include/python3.5m/ -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCUDA_ARCH=35 . && make

And the result which points to another error.

-- Configuring done -- Generating done -- Build files have been written to: /home/qxw41/tools/kmcuda/src [ 14%] Building CXX object CMakeFiles/KMCUDA.dir/kmcuda.cc.o /home/qxw41/tools/kmcuda/src/kmcuda.cc:206:0: error: ignoring #pragma omp simd [-Werror=unknown-pragmas]

pragma omp simd

^ /home/qxw41/tools/kmcuda/src/kmcuda.cc:310:0: error: ignoring #pragma omp simd [-Werror=unknown-pragmas]

pragma omp simd reduction(+:dist_sum2)

^ cc1plus: all warnings being treated as errors make[2]: [CMakeFiles/KMCUDA.dir/kmcuda.cc.o] Error 1 make[1]: [CMakeFiles/KMCUDA.dir/all] Error 2 make: *** [all] Error 2

My working environment is: NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Thanks.

vmarkovtsev commented 6 years ago

Centos 7 has gcc 4.8 which is too old - it does not support OpenMP 4.0. You need to upgrade your Compiler.

QinglongWang commented 6 years ago

Thanks, which version of gcc should be OK? GCC 5 or 6? I could try to install it

vmarkovtsev commented 6 years ago

Funnily, any version which is newer than 4.8.

QinglongWang commented 6 years ago

I use this link to upgrade gcc to: gcc (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)

But still I have following error:

-- Configuring done -- Generating done -- Build files have been written to: /home/qxw41/tools/kmcuda/src [ 14%] Building CXX object CMakeFiles/KMCUDA.dir/kmcuda.cc.o /home/qxw41/tools/kmcuda/src/kmcuda.cc:206:0: error: ignoring #pragma omp simd [-Werror=unknown-pragmas]

pragma omp simd

^ /home/qxw41/tools/kmcuda/src/kmcuda.cc:310:0: error: ignoring #pragma omp simd [-Werror=unknown-pragmas]

pragma omp simd reduction(+:dist_sum2)

^ cc1plus: all warnings being treated as errors CMakeFiles/KMCUDA.dir/build.make:778: recipe for target 'CMakeFiles/KMCUDA.dir/kmcuda.cc.o' failed make[2]: [CMakeFiles/KMCUDA.dir/kmcuda.cc.o] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/KMCUDA.dir/all' failed make[1]: [CMakeFiles/KMCUDA.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: *** [all] Error 2

Is there anyway to correct this? Thank you very much.

vmarkovtsev commented 6 years ago

What does sudo yum list devtoolset-6\* say? which g++? g++ --version?

QinglongWang commented 6 years ago

g++ --version gives me: g++ (GCC) 6.3.1 20170216 (Red Hat 6.3.1-3)

sudo yum list devtoolset-6*

Loaded plugins: fastestmirror, keys Loading mirror speeds from cached hostfile

Installed Packages devtoolset-6.x86_64 6.1-1.el7 @centos-sclo-rh devtoolset-6-binutils.x86_64 2.27-10.el7 @centos-sclo-rh devtoolset-6-dwz.x86_64 0.12-1.el7 @centos-sclo-rh devtoolset-6-dyninst.x86_64 9.2.0-4.el7 @centos-sclo-rh devtoolset-6-elfutils.x86_64 0.168-3.el7 @centos-sclo-rh devtoolset-6-elfutils-libelf.x86_64 0.168-3.el7 @centos-sclo-rh devtoolset-6-elfutils-libs.x86_64 0.168-3.el7 @centos-sclo-rh devtoolset-6-gcc.x86_64 6.3.1-3.1.el7 @centos-sclo-rh devtoolset-6-gcc-c++.x86_64 6.3.1-3.1.el7 @centos-sclo-rh devtoolset-6-gcc-gfortran.x86_64 6.3.1-3.1.el7 @centos-sclo-rh devtoolset-6-gdb.x86_64 7.12.1-47.el7 @centos-sclo-rh devtoolset-6-libquadmath-devel.x86_64 6.3.1-3.1.el7 @centos-sclo-rh devtoolset-6-libstdc++-devel.x86_64 6.3.1-3.1.el7 @centos-sclo-rh devtoolset-6-ltrace.x86_64 0.7.91-17.el7 @centos-sclo-rh devtoolset-6-make.x86_64 1:4.1-3.el7 @centos-sclo-rh devtoolset-6-memstomp.x86_64 0.1.5-5.el7 @centos-sclo-rh devtoolset-6-oprofile.x86_64 1.1.0-4.el7 @centos-sclo-rh devtoolset-6-perftools.x86_64 6.1-1.el7 @centos-sclo-rh devtoolset-6-runtime.x86_64 6.1-1.el7 @centos-sclo-rh devtoolset-6-strace.x86_64 4.12-3.el7 @centos-sclo-rh devtoolset-6-systemtap.x86_64 3.0-8s.el7 @centos-sclo-rh devtoolset-6-systemtap-client.x86_64 3.0-8s.el7 @centos-sclo-rh devtoolset-6-systemtap-devel.x86_64

vmarkovtsev commented 6 years ago

cmake could cache the path to g++. Remove the build directory completely and repeat. If there is no build directory, git clean -xfd

abhishek-shukla-93 commented 6 years ago

Hello,

I am trying to set up this project to use K-man GPU implementation in Python. In set up steps, I am getting Error while giving the

command: pip install git+https://github.com/src-d/kmcuda.git#subdirectory=src

Error: Collecting git+https://github.com/src-d/kmcuda.git#subdirectory=src Cloning https://github.com/src-d/kmcuda.git to /tmp/pip-NGyzQG-build Requirement already satisfied: numpy in /usr/local/lib64/python2.7/site-packages (from libKMCUDA==6.2.1) Installing collected packages: libKMCUDA Running setup.py install for libKMCUDA ... error Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-NGyzQG-build/src/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-PJz8TS-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_py -- The C compiler identification is GNU 4.8.3 -- The CXX compiler identification is GNU 4.8.3 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Try OpenMP C flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Success -- Try OpenMP CXX flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Success -- Found OpenMP: -fopenmp -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found PythonInterp: /usr/bin/python3 (found suitable version "3.4.3", minimum required is "3") -- Found PythonLibs: /usr/lib64/libpython3.4m.so (found suitable version "3.4.3", minimum required is "3") -- Configuring done -- Generating done -- Build files have been written to: /tmp/pip-NGyzQG-build/src [ 16%] Building NVCC (Device) object CMakeFiles/KMCUDA.dir/KMCUDA_generated_transpose.cu.o [ 33%] Building NVCC (Device) object CMakeFiles/KMCUDA.dir/KMCUDA_generated_kmeans.cu.o [ 50%] Building NVCC (Device) object CMakeFiles/KMCUDA.dir/KMCUDA_generated_knn.cu.o nvcc fatal : Value 'sm_61' is not defined for option 'gpu-architecture' nvcc fatal : Value 'sm_61' is not defined for option 'gpu-architecture' nvcc fatal : Value 'sm_61' is not defined for option 'gpu-architecture' CMake Error at KMCUDA_generated_transpose.cu.o.Release.cmake:207 (message): Error generating /tmp/pip-NGyzQG-build/src/CMakeFiles/KMCUDA.dir//./KMCUDA_generated_transpose.cu.o

CMake Error at KMCUDA_generated_kmeans.cu.o.Release.cmake:207 (message):
  Error generating
  /tmp/pip-NGyzQG-build/src/CMakeFiles/KMCUDA.dir//./KMCUDA_generated_kmeans.cu.o

CMake Error at KMCUDA_generated_knn.cu.o.Release.cmake:207 (message):
  Error generating
  /tmp/pip-NGyzQG-build/src/CMakeFiles/KMCUDA.dir//./KMCUDA_generated_knn.cu.o

make[2]: *** [CMakeFiles/KMCUDA.dir/KMCUDA_generated_transpose.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/KMCUDA.dir/KMCUDA_generated_kmeans.cu.o] Error 1
make[2]: *** [CMakeFiles/KMCUDA.dir/KMCUDA_generated_knn.cu.o] Error 1
make[1]: *** [CMakeFiles/KMCUDA.dir/all] Error 2
make: *** [all] Error 2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-NGyzQG-build/src/setup.py", line 80, in <module>
    "Programming Language :: Python :: 3.6",
  File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib64/python2.7/distutils/command/install.py", line 604, in run
    self.run_command('build')
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/pip-NGyzQG-build/src/setup.py", line 19, in run
    self._build()
  File "/tmp/pip-NGyzQG-build/src/setup.py", line 44, in _build
    check_call(("make", "-j%d" % cpu_count()))
  File "/usr/lib64/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('make', '-j8')' returned non-zero exit status 2

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-NGyzQG-build/src/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-PJz8TS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-NGyzQG-build/src

Can anybody please help me to resolve this issue. I am using cuda-7.5

Thanks,

vmarkovtsev commented 6 years ago

Hi @abhishek-mascon

Can you please report a separate issue? I know the answer but don't want to mess up this issue even more. Actually, closing it as there was no response from the original author.

abhishek-shukla-93 commented 6 years ago

@vmarkovtsev : Thanks for your response. Opened it as separate issue #44 .