Compile custom operator failed in Detectron

[7oud]

Detailed steps to reproduce

1. Compile caffe2 source code and install it into conda environment
2. Test caffe2 OK, when running python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'
3. Install other dependencies and detectron
4. Detectron tests pass

   
   I0307 21:53:58.443390 23369 operator.cc:173] Operator with engine CUDNN is not available for operator SpatialNarrowAsGradient.
   ----------------------------------------------------------------------
   Ran 3 tests in 0.805s

OK

5. Complie custom op **FAILED**, I am not sure it is the problem of my env or a bug in detectron or caffe2, it seems some similar situation [cocodataset/cocoapi#94 (comment)](https://github.com/cocodataset/cocoapi/issues/94#issuecomment-350152998)

(caffe2) zoud@ubu1604:~/Workspace/Github/facebookresearch/detectron/lib$ make ops mkdir -p build && cd build && cmake .. && make -j4 -- Caffe2: Found gflags with new-style gflags target. -- Caffe2: Cannot find glog automatically. Using legacy find. ... -- CUDA detected: 8.0 CMake Error at cmake/Cuda.cmake:32 (message): Invalid CUDA_ARCH_NAME, supported values: Kepler, Maxwell, Pascal, Volta, All, Manual. Got Auto Call Stack (most recent call first): cmake/Cuda.cmake:183 (caffe2_select_nvcc_arch_flags) cmake/Dependencies.cmake:11 (include) CMakeLists.txt:14 (include)

-- Configuring incomplete, errors occurred! See also "/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeOutput.log". See also "/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeError.log". Makefile:13: recipe for target 'ops' failed make: *** [ops] Error 1

**ERROR** info in CMakeError.log, it seems pthread can not be found

Linking C executable cmTC_ec809 /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_ec809.dir/link.txt --verbose=1 /usr/bin/cc -std=c11 CMakeFiles/cmTC_ec809.dir/CheckSymbolExists.c.o -o cmTC_ec809 -rdynamic CMakeFiles/cmTC_ec809.dir/CheckSymbolExists.c.o：在函数‘main’中： CheckSymbolExists.c:(.text+0x16)：对‘pthread_create’未定义的引用 collect2: error: ld returned 1 exit status CMakeFiles/cmTC_ec809.dir/build.make:97: recipe for target 'cmTC_ec809' failed make[2]: [cmTC_ec809] Error 1 make[2]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' Makefile:126: recipe for target 'cmTC_ec809/fast' failed make[1]: [cmTC_ec809/fast] Error 2 make[1]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp'

File /home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c: / /

include

int main(int argc, char** argv) { (void)argv;

ifndef pthread_create

return ((int*)(&pthread_create))[argc];

else

(void)argc; return 0;

endif

}

Determining if the function pthread_create exists in the pthreads failed with the following output: Change Dir: /home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_761f6/fast" make[1]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' /usr/bin/make -f CMakeFiles/cmTC_761f6.dir/build.make CMakeFiles/cmTC_761f6.dir/build make[2]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' Building C object CMakeFiles/cmTC_761f6.dir/CheckFunctionExists.c.o /usr/bin/cc -std=c11 -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_761f6.dir/CheckFunctionExists.c.o -c /usr/share/cmake-3.5/Modules/CheckFunctionExists.c Linking C executable cmTC_761f6 /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_761f6.dir/link.txt --verbose=1 /usr/bin/cc -std=c11 -DCHECK_FUNCTION_EXISTS=pthread_create CMakeFiles/cmTC_761f6.dir/CheckFunctionExists.c.o -o cmTC_761f6 -rdynamic -lpthreads /usr/bin/ld: 找不到 -lpthreads collect2: error: ld returned 1 exit status CMakeFiles/cmTC_761f6.dir/build.make:97: recipe for target 'cmTC_761f6' failed make[2]: [cmTC_761f6] Error 1 make[2]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' Makefile:126: recipe for target 'cmTC_761f6/fast' failed make[1]: [cmTC_761f6/fast] Error 2 make[1]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp'

**ERROR** info in CMakeOutput.log

Determining if the include file pthread.h exists passed with the following output: Change Dir: /home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_c8ca7/fast" make[1]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' /usr/bin/make -f CMakeFiles/cmTC_c8ca7.dir/build.make CMakeFiles/cmTC_c8ca7.dir/build make[2]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' Building C object CMakeFiles/cmTC_c8ca7.dir/CheckIncludeFile.c.o /usr/bin/cc -std=c11 -o CMakeFiles/cmTC_c8ca7.dir/CheckIncludeFile.c.o -c /home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp/CheckIncludeFile.c Linking C executable cmTC_c8ca7 /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c8ca7.dir/link.txt --verbose=1 /usr/bin/cc -std=c11 CMakeFiles/cmTC_c8ca7.dir/CheckIncludeFile.c.o -o cmTC_c8ca7 -rdynamic make[2]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' make[1]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp'

Determining if the function pthread_create exists in the pthread passed with the following output: Change Dir: /home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/make" "cmTC_82b20/fast" make[1]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' /usr/bin/make -f CMakeFiles/cmTC_82b20.dir/build.make CMakeFiles/cmTC_82b20.dir/build make[2]: Entering directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' Building C object CMakeFiles/cmTC_82b20.dir/CheckFunctionExists.c.o /usr/bin/cc -std=c11 -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_82b20.dir/CheckFunctionExists.c.o -c /usr/share/cmake-3.5/Modules/CheckFunctionExists.c Linking C executable cmTC_82b20 /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_82b20.dir/link.txt --verbose=1 /usr/bin/cc -std=c11 -DCHECK_FUNCTION_EXISTS=pthread_create CMakeFiles/cmTC_82b20.dir/CheckFunctionExists.c.o -o cmTC_82b20 -rdynamic -lpthread make[2]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp' make[1]: Leaving directory '/home/zoud/Workspace/Github/facebookresearch/detectron/lib/build/CMakeFiles/CMakeTmp'

### System information

* Operating system: Ubuntu16.04
* Compiler version: gcc 5.4.0
* CUDA version: 8.0
* cuDNN version: 7.0.5
* NVIDIA driver version: 384.111
* GPU models (for all devices if they are not all the same): GTX 780Ti
* `PYTHONPATH` environment variable: 
* `python --version` output: Python 2.7.14 :: Anaconda, Inc. (conda environment)

[orionr]

Looks like the auto detection is failing. Can you try building with

mkdir -p build && cd build && cmake .. -DCUDA_ARCH_NAME=Kepler && make -j4

and see if that fixes things? This is because you have a GTX 780Ti, which I think is Kepler chipset.

[fengyicoder]

@orionr I try to use your command to resolve this problem, but the error appear and is as following: [ 20%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o [ 60%] Built target caffe2_detectron_custom_ops nvcc fatal : redefinition of argument 'std' CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:203 (message): Error generating /home/flyfree/detectron/lib/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o' failed make[2]: [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o] Error 1 CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all' failed make[1]: [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all] Error 2 Makefile:127: recipe for target 'all' failed make: *** [all] Error 2

[sdalxn]

@orionr My Graphic card is Titan X Pascal. How can I specify the CUDA_ARCH_NAME？ Is it also Kepler in this statement? mkdir -p build && cd build && cmake .. -DCUDA_ARCH_NAME=Kepler && make -j4

[orionr]

@sdalxn for a Titan X Pascal you'd want mkdir -p build && cd build && cmake .. -DCUDA_ARCH_NAME=Pascal && make -j4. Note that you shouldn't need to typically specify this and should only do so if you see the error above. Hope that helps.

[sdalxn]

@orionr When I install caffe2, I have no any error. When I install dectron, I have no error. But when I Build the custom operators library as "cd $DETECTRON/lib && make ops"， I got ”CMake Error at cmake/Cuda.cmake:32 (message): Invalid CUDA_ARCH_NAME, supported values: Kepler, Maxwell, Pascal, Volta, All, Manual. Got Auto“

It is same as https://github.com/facebookresearch/Detectron/issues/251 I mentioned your advice on this issue. So I want to know how to make caffe2 for detectron.

[jinwu07]

@orionr It doesn't fix the problem completely. It encountered a different error after.

[ 40%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/ops/zero_even_op.cc.o nvcc fatal : redefinition of argument 'std' CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:203 (message): Error generating /home/jinwu/ml/detectron/lib/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

This is the make command line: mkdir -p build && cd build && cmake .. -DCUDA_ARCH_NAME=Pascal && make -j$(shell nproc)

[jinwu07]

@orionr @nangongtianyi By following the thread https://github.com/facebookresearch/Detectron/issues/185, I was able to get around the problem.

Comment out the line in Cuda.cmake: list(APPEND CUDA_NVCC_FLAGS "-std=c++11")

[yinghai]

Yeah, we probably need something like

if(NOT "${CUDA_NVCC_FLAGS}" MATCHES "-std=c\\+\\+11" )
  list(APPEND CUDA_NVCC_FLAGS -std=c++11)
endif()

[ir413]

Addressed by facebookresearch/Detectron#286.