getting the following error when building nimble in conda enviornment FAILED: caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.o subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 1.

umairjavaid commented 3 years ago

[46/402] Building NVCC (Device) object...ils/torch_cuda_generated_math_gpu.cu.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.o 
cd /home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils && /home/umair/anaconda3/envs/nimble/bin/cmake -E make_directory /home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils/. && /home/umair/anaconda3/envs/nimble/bin/cmake -D verbose:BOOL=OFF -D build_configuration:STRING=Release -D generated_file:STRING=/home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.o -D generated_cubin_file:STRING=/home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.o.cubin.txt -P /home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils/torch_cuda_generated_math_gpu.cu.o.Release.cmake
/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(149): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(196): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(231): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(898): error: namespace "thrust" has no member "host_vector"

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(898): error: expected an expression

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(899): error: namespace "thrust" has no member "host_vector"

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(899): error: expected an expression

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(900): error: namespace "thrust" has no member "host_vector"

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(900): error: type name is not allowed

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(900): error: expected an expression

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(902): error: identifier "A_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(903): error: identifier "B_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(904): error: identifier "C_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(907): error: identifier "A_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(909): error: identifier "B_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(910): error: identifier "C_array" is undefined

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(1749): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(2211): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(2258): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu(2814): warning: the "__visibility__" attribute can only appear on functions and variables with external linkage

13 errors detected in the compilation of "/home/umair/Desktop/umair/nimble2/nimble/caffe2/utils/math_gpu.cu".
CMake Error at torch_cuda_generated_math_gpu.cu.o.Release.cmake:281 (message):
  Error generating file
  /home/umair/Desktop/umair/nimble2/nimble/build/caffe2/CMakeFiles/torch_cuda.dir/utils/./torch_cuda_generated_math_gpu.cu.o

[53/402] Building NVCC (Device) object...cuda_generated_elementwise_div_op.cu.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 760, in <module>
    build_deps()
  File "setup.py", line 315, in build_deps
    cmake=cmake)
  File "/home/umair/Desktop/umair/nimble2/nimble/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/umair/Desktop/umair/nimble2/nimble/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/umair/Desktop/umair/nimble2/nimble/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/umair/anaconda3/envs/nimble/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 1.

gyeongin commented 3 years ago

Looks like this error is related to this: https://github.com/pytorch/pytorch/issues/61011. Did you use CUDA 11.4? If so, please retry with CUDA 11.0.

umairjavaid commented 3 years ago

I have three directories: cuda, cuda-11 and cuda 11.4 on my system. I am giving the path to cuda-11 in my case.

gyeongin commented 3 years ago

The error does not occur with CUDA v11.0, so there must be some problem with your CUDA setting (perhaps missing environment variables?). You can check the log to see which version of CUDA you are using. The build log prints the following information at the beginning:

...
...
...
--   USE_CUDA              : ON
--     CUDA static link    : OFF
--     USE_CUDNN           : ON
--     CUDA version        : 11.0
--     cuDNN version       : 8.0.5
--     CUDA root directory : /home/gyeongin/cuda-11.0
...
...
...

emjay73 commented 2 years ago

I added the following to caffe2/utils/math_gpu.cu

#include <thrust/host_vector.h>

and it worked like charm.

ref : https://github.com/pytorch/pytorch/issues/61011

snuspl / nimble