ERROR : Run demo.py --gpu 0

jhj7905 commented 7 years ago

Spec : cuda8.0 cudnn5.1

Errors occured below

cudnn_cov_lay.cu:33] check failed: status == CUDNN_STATUS_SUCCESS( 5 vs. 0 ) CUDN_STATUS_INVALID_VALUE

how can i run demo.py by gpu

Cysu commented 7 years ago

@jhj7905 Could you please attach the full error report?

jhj7905 commented 7 years ago

@Cysu i attached the error report below

thank you

I0830 11:22:04.826299 8202 net.cpp:1272] blob 169 name det_score diff idx -1 I0830 11:22:04.826318 8202 net.cpp:1262] blob 170 name pid_prob data idx -1 I0830 11:22:04.826335 8202 net.cpp:1272] blob 170 name pid_prob diff idx -1 I0830 11:22:04.826354 8202 net.cpp:1262] blob 171 name bbox_pred data idx -1 I0830 11:22:04.826371 8202 net.cpp:1272] blob 171 name bbox_pred diff idx -1 I0830 11:22:04.826390 8202 net.cpp:1262] blob 172 name feat_lowdim data idx 0 I0830 11:22:04.826409 8202 net.cpp:1272] blob 172 name feat_lowdim diff idx 0 I0830 11:22:04.826427 8202 net.cpp:1262] blob 173 name feat data idx -1 I0830 11:22:04.826445 8202 net.cpp:1272] blob 173 name feat diff idx -1 I0830 11:22:04.826462 8202 net.cpp:1278] storage memory slot 0 size 3211264 I0830 11:22:04.826479 8202 net.cpp:1278] storage memory slot 1 size 3211264 I0830 11:22:04.826496 8202 net.cpp:1278] storage memory slot 2 size 3211264 I0830 11:22:04.826514 8202 net.cpp:1278] storage memory slot 3 size 401408 I0830 11:22:04.826530 8202 net.cpp:1283] raw memory 310126112 opt memory 11241632 I0830 11:22:34.060151 8202 cudnn_conv_layer.cpp:186] Optimized cudnn conv demo/image2.jpg ... F0830 11:22:34.267608 8202 cudnn_conv_layer.cu:33] Check failed: status == CUDNN_STATUS_SUCCESS (5 vs. 0) CUDNN_STATUS_INVALID_VALUE Check failure stack trace: [ubuntu:08202] Process received signal [ubuntu:08202] Signal: Aborted (6) [ubuntu:08202] Signal code: (-6) [ubuntu:08202] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7ff597ba5330] [ubuntu:08202] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7ff597802c37] [ubuntu:08202] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7ff597806028] [ubuntu:08202] [ 3] /usr/local/lib/libglog.so.0(+0xe799) [0x7ff528486799] [ubuntu:08202] [ 4] /usr/local/lib/libglog.so.0(+0xfebd) [0x7ff528487ebd] [ubuntu:08202] [ 5] /usr/local/lib/libglog.so.0(_ZN6google10LogMessage9SendToLogEv+0x258) [0x7ff528489e08] [ubuntu:08202] [ 6] /usr/local/lib/libglog.so.0(_ZN6google10LogMessage5FlushEv+0xa3) [0x7ff528487a53] [ubuntu:08202] [ 7] /usr/local/lib/libglog.so.0(_ZN6google15LogMessageFatalD1Ev+0xe) [0x7ff52848a73e] [ubuntu:08202] [ 8] /home/jhj7905/person_search-master/caffe/build/lib/libcaffe.so(_ZN5caffe21CuDNNConvolutionLayerIfE11Forward_gpuERKSt6vectorIPNS_4BlobIfEESaIS5EES9+0x45e) [0x7ff529aee31e] [ubuntu:08202] [ 9] /home/jhj7905/person_search-master/caffe/build/lib/libcaffe.so(_ZN5caffe3NetIfE13ForwardFromToEii+0x135) [0x7ff529aad385] [ubuntu:08202] [10] /home/jhj7905/person_search-master/tools/../caffe/python/caffe/_caffe.so(_ZN5boost6python7objects23caller_py_function_implINS0_6detail6callerIMN5caffe3NetIfEEFfiiENS0_21default_call_policiesENS_3mpl7vector4IfRS7_iiEEEEEclEP7objectSI+0xf9) [0x7ff529f43d49] [ubuntu:08202] [11] /usr/local/lib/libboost_python.so.1.59.0(_ZNK5boost6python7objects8function4callEP7objectS4+0xca) [0x7ff5290adfaa] [ubuntu:08202] [12] /usr/local/lib/libboost_python.so.1.59.0(+0x29318) [0x7ff5290ae318] [ubuntu:08202] [13] /usr/local/lib/libboost_python.so.1.59.0(_ZN5boost6python21handle_exception_implENS_9function0IvEE+0x53) [0x7ff5290b8623] [ubuntu:08202] [14] /usr/local/lib/libboost_python.so.1.59.0(+0x27c23) [0x7ff5290acc23] [ubuntu:08202] [15] python(PyEval_EvalFrameEx+0x98d) [0x5244dd] [ubuntu:08202] [16] python() [0x568b3a] [ubuntu:08202] [17] python(PyEval_EvalFrameEx+0x2167) [0x525cb7] [ubuntu:08202] [18] python(PyEval_EvalCodeEx+0x2b1) [0x555551] [ubuntu:08202] [19] python(PyEval_EvalFrameEx+0x7e8) [0x524338] [ubuntu:08202] [20] python(PyEval_EvalCodeEx+0x2b1) [0x555551] [ubuntu:08202] [21] python(PyEval_EvalFrameEx+0x7e8) [0x524338] [ubuntu:08202] [22] python(PyEval_EvalFrameEx+0xc9a) [0x5247ea] [ubuntu:08202] [23] python() [0x567d14] [ubuntu:08202] [24] python(PyRun_FileExFlags+0x92) [0x465bf4] [ubuntu:08202] [25] python(PyRun_SimpleFileExFlags+0x2ee) [0x46612d] [ubuntu:08202] [26] python(Py_Main+0xb5e) [0x466d92] [ubuntu:08202] [27] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ff5977edf45] [ubuntu:08202] [28] python() [0x577c2e] [ubuntu:08202] End of error message Aborted (core dumped)

Cysu commented 7 years ago

@jhj7905 I have checked the cudnn v5.1 manual and there was no CUDNN_STATUS_INVALID_VALUE error type relavent to convolution operations.

Could you please check the output of the following command:

ldd caffe/build/install/bin/caffe | grep cudnn

Also note that cudnn v5.1 has cuda-7.5 and cuda-8.0 versions. Please make sure the correct version is installed and linked.

jhj7905 commented 7 years ago

@Cysu there is no such file or direcotry when i used ldd caffe/build/install/bin/caffe | grep cudnn...

Cysu commented 7 years ago

@jhj7905 How did you compile the caffe? Did you follow the cmake commands listed in our README?

jhj7905 commented 7 years ago

@Cysu Can you show me your cmakelist when you compiled the caffe...

jhj7905 commented 7 years ago

@Cysu I recompiled the caffe by modifing the cmakelists. there is libcudnn.so.5 when i used ldd caffe/build/install/bin/caffe | grep cudnn... i want to confirm your cmakelists..please show your one???

Cysu commented 7 years ago

@jhj7905 I didn't modify the CMakeLists.txt. Usually it is configured through command line parameters, like the one we shown in the README:

cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/path/to/cudnn/include -DCUDNN_LIBRARY=/path/to/cudnn/lib64/libcudnn.so

Here what I mean is that please make sure the cudnn it linked to is for cuda-8.0, not cuda-7.5. There are two versions of cudnn-5.1 on the nvidia website for download.

jhj7905 commented 7 years ago

@Cysu Do you mean that It is not correct (there is libcudnn.so.5 when i used ldd caffe/build/install/bin/caffe | grep cudnn...)?

i installed cuda-8.0, cudnn-5.1...but still same result like below cudnn_cov_lay.cu:33] check failed: status == CUDNN_STATUS_SUCCESS( 5 vs. 0 ) CUDN_STATUS_INVALID_VALUE

Could you tell me how to solve the problem in details?

Cysu commented 7 years ago

The libcudnn.so.5 normally links to libcudnn.so.5.1.10. Could you please check the file size of the libcudnn.so.5.1.10? It should be 84163560 bytes. If not, the version is probably not correct.

jhj7905 commented 7 years ago

@Cysu I checked the file size of libcudnn.so.5.1.10... like below -rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so -rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so.5 -rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so.5.1.10* -rw-r--r-- 1 root root 70364814 Aug 30 09:32 libcudnn_static.a hmmm...

Cysu commented 7 years ago

Alright, that is correct. I wonder if it is due to out of memory? Could you please check the memory consumption with nvidia-smi?

jhj7905 commented 7 years ago

jhj7905 commented 7 years ago

cuz..i did not install openmpi... because of python termination

Cysu commented 7 years ago

@jhj7905 It seems that the GPU 0 is almost occupied (9GB / 12GB). You may try to set --gpu 1 and see if it works.

To install openmpi, please download the source from here, then

tar xf openmpi-1.10.7.tar.gz
cd openmpi-1.10.7
./configure --with-cuda=/usr/local/cuda --enable-mpi-thread-multiple
make -j8
sudo make install
cd -

This will by default install it to /usr/local/. To use it, please add the following line to your ~/.bashrc

export PATH=/usr/local/bin:$PATH

Restart the terminal, remove the caffe/build directory, and compile it again with -DUSE_MPI=ON option in cmake.

jhj7905 commented 7 years ago

@Cysu I did it as you told me..set --gpu 1..but still did not work...... hmm.....what is your pc spec??... i have questions about your network. you use a softmax classifier whether the region is person or not right? and.. usually the output of bbox is 4..but you have 8 bbox..why? and what does it consist?(for example, x1,x2?? ) can you tell me in details?

jhj7905 commented 7 years ago

@Cysu i have a problem..when i build with mpi like below hmm

jhj7905@ubuntu:~/person_search-master$ python tools/demo.py [ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_paffinity_hwloc: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored) [ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored) [ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_carto_file: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored) [ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_shmem_posix: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored) [ubuntu:37345] Process received signal [ubuntu:37345] Signal: Segmentation fault (11) [ubuntu:37345] Signal code: Address not mapped (1) [ubuntu:37345] Failing at address: (nil) [ubuntu:37345] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f1c08b04330] [ubuntu:37345] [ 1] /usr/local/lib/libopen-pal.so.13(+0x46dcb) [0x7f1be200bdcb] [ubuntu:37345] End of error message Segmentation fault (core dumped)

Cysu commented 7 years ago

@jhj7905 Oh, I forgot to mention that you may need to also add the following line to ~/.bashrc:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

By the way, could you please also verify that which mpirun refers to /usr/local/bin/mpirun and mpirun --version report 1.10.7?

Cysu commented 7 years ago

@jhj7905 My PC is Titan X Haswell, cuda-8, cudnn-v5.1.

The reason why we have 8 output units for bboxes instead of 4 is that the original py-faster-rcnn implementation was for general object detection, say there could be 20 object classes + 1 background class. Thus they have in total 21 bboxes, one for each class. We inherit this part of code, so there are 2 bboxes for pedestrian and non-pedestrian. The one for non-pedestrian is just for simplicity and has no effect in practice.

jhj7905 commented 7 years ago

@Cysu Sincerly thank you for replying my questions. I understandd the reason why you have 8 output units for bboexs.. anyway.. my statue about openmpi like below jhj7905@ubuntu:~$ which mpirun /usr/local/bin/mpirun jhj7905@ubuntu:~$ mpirun --version mpirun: /usr/local/cuda/lib64/libOpenCL.so.1: no version information available (required by /usr/local/lib/libopen-pal.so.13) mpirun (Open MPI) 1.10.7

Report bugs to http://www.open-mpi.org/community/help/

above case, when i type 'from mpi4py import MPI', then error occur

one more thing. could u tell me how to use the multi-gpu

i should run it by using gpu...cuz you have given me a lot of support.. thank u

Cysu commented 7 years ago

@jhj7905 You mean it's fine to run the demo with one GPU now? That's great.

If you haven't installed the mpi4py package before, you can install it with

pip install mpi4py

The demo is not for multi-gpu. We currently only have the evaluation script supporting multi-gpu. Sorry about the inconvenience.

jhj7905 commented 7 years ago

@Cysu I have a question about building the mpi.. do u use opencl when build it?

Cysu commented 7 years ago

@jhj7905 No, I didn't use opencl when building mpi. I used exactly the same commands as I listed above.

liuajian commented 7 years ago

@jhj7905 unbuntu 16.04 + cuda8.0+cudnn5.1就可以了， openmpi不用安装系统自带的就是openmpi-1.10.2 直接按作者的cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/path/to/cudnn/include -DCUDNN_LIBRARY=/path/to/cudnn/lib64/libcudnn.so 就可以了

XinshaoWang commented 7 years ago

@liuajian How do you make it? I have installed all the dependencies as the author list. My work environment is Ubuntu 16.04+cuda8.0. When I used the cudnn v6.0, the errors are as follows:

-- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
-- Boost version: 1.58.0 -- Found the following Boost libraries: -- system -- thread -- chrono -- date_time -- atomic -- Found GFlags: /usr/include
-- Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so) -- Found Glog: /usr/include
-- Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so) -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found PROTOBUF Compiler: /usr/bin/protoc -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.8.16") -- Found LMDB: /usr/include
-- Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so) -- Found LevelDB: /usr/include
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so) -- Found Snappy: /usr/include
-- Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so) -- CUDA detected: 8.0 -- Found cuDNN (include: /usr/local/cuda-8.0/include, library: /usr/local/cuda-8.0/lib64/libcudnn.so) -- Added CUDA NVCC flags for: sm_61 -- OpenCV found (/usr/local/share/OpenCV) -- Found Atlas: /usr/include
-- Found Atlas (include: /usr/include, library: /usr/lib/libatlas.so) -- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7") -- Found NumPy: /usr/lib/python2.7/dist-packages/numpy/core/include (found suitable version "1.11.0", minimum required is "1.7.1") -- NumPy ver. 1.11.0 found (include: /usr/lib/python2.7/dist-packages/numpy/core/include) -- Boost version: 1.58.0 -- Found the following Boost libraries: -- python -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) -- Found MPI_C: /usr/local/lib/libmpi.so
-- Found MPI_CXX: /usr/local/lib/libmpi_cxx.so;/usr/local/lib/libmpi.so
-- Found Git: /usr/bin/git (found version "2.7.4") -- -- Caffe Configuration Summary -- General: -- Version : (Caffe doesn't declare its version in headers) -- Git : v0.9999-1625-gaed3884-dirty -- System : Linux -- C++ compiler : /usr/bin/c++ -- Release CXX flags : -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized -- Debug CXX flags : -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized -- Build type : Release -- -- BUILD_SHARED_LIBS : ON -- BUILD_python : ON -- BUILD_matlab : OFF -- BUILD_docs : ON -- CPU_ONLY : OFF -- -- Dependencies: -- BLAS : Yes (Atlas) -- Boost : Yes (ver. 1.58) -- glog : Yes -- gflags : Yes -- protobuf : Yes (ver. 2.6.1) -- lmdb : Yes (ver. 0.9.17) -- Snappy : Yes (ver. 1.1.3) -- LevelDB : Yes (ver. 1.18) -- OpenCV : Yes (ver. 3.1.0) -- CUDA : Yes (ver. 8.0) -- -- NVIDIA CUDA: -- Target GPU(s) : Auto -- GPU arch(s) : sm_61 -- cuDNN : Yes -- -- Python: -- Interpreter : /usr/bin/python2.7 (ver. 2.7.12) -- Libraries : /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12) -- NumPy : /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0) -- -- Documentaion: -- Doxygen : No -- config_file :
-- -- Install: -- Install path : /home/amos/person_search/caffe/build/install -- -- Configuring done -- Generating done -- Build files have been written to: /home/amos/person_search/caffe/build [ 0%] Running C++/Python protocol buffer compiler on /home/amos/person_search/caffe/src/caffe/proto/caffe.proto Scanning dependencies of target proto [ 1%] Building CXX object src/caffe/CMakeFiles/proto.dir///include/caffe/proto/caffe.pb.cc.o [ 1%] Linking CXX static library ../../lib/libproto.a [ 1%] Built target proto [ 2%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o [ 2%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o [ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000312_00000000-5_im2col.cpp4.ii". CMake Error at cuda_compile_generated_im2col.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_im2col.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:70: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o' failed make[2]: [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o] Error 1 make[2]: Waiting for unfinished jobs.... /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000031f_00000000-5_labeled_matching_layer.cpp4.ii". CMake Error at cuda_compile_generated_labeled_matching_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_labeled_matching_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:91: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000339_00000000-5_absval_layer.cpp4.ii". /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

CMake Error at cuda_compile_generated_absval_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_absval_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:77: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000033e_00000000-5_pooling_layer.cpp4.ii". 1 error detected in the compilation of "/tmp/tmpxft_0000034a_00000000-5_cudnn_conv_layer.cpp4.ii". CMake Error at cuda_compile_generated_pooling_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_pooling_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:105: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o] Error 1 CMake Error at cuda_compile_generated_cudnn_conv_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_conv_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:98: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o] Error 1 1 error detected in the compilation of "/tmp/tmpxft_00000352_00000000-5_concat_layer.cpp4.ii". CMake Error at cuda_compile_generated_concat_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_concat_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:84: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000331_00000000-5_math_functions.cpp4.ii". CMake Error at cuda_compile_generated_math_functions.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:63: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o] Error 1 1 error detected in the compilation of "/tmp/tmpxft_0000035a_00000000-5_cudnn_softmax_layer.cpp4.ii". CMake Error at cuda_compile_generated_cudnn_softmax_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_softmax_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:427: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o' failed make[2]: [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o] Error 1 CMakeFiles/Makefile2:240: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed make[1]: [src/caffe/CMakeFiles/caffe.dir/all] Error 2 Makefile:127: recipe for target 'all' failed make: *** [all] Error 2

XinshaoWang commented 7 years ago

@liuajian @Cysu When I used cudnn v5.1, the errors are as follows: ~/person_search/caffe/build$ cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/home/amos/cudnn-8.0-linux-x64-v5.1/include -DCUDNN_LIBRARY=/home/amos/cudnn-8.0-linux-x64-v5.1/lib64/libcudnn.so -- The C compiler identification is GNU 5.4.0 -- The CXX compiler identification is GNU 5.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
-- Boost version: 1.58.0 -- Found the following Boost libraries: -- system -- thread -- chrono -- date_time -- atomic -- Found GFlags: /usr/include
-- Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so) -- Found Glog: /usr/include
-- Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so) -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found PROTOBUF Compiler: /usr/bin/protoc -- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.8.16") -- Found LMDB: /usr/include
-- Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so) -- Found LevelDB: /usr/include
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so) -- Found Snappy: /usr/include
-- Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so) -- CUDA detected: 8.0 -- Found cuDNN (include: /home/amos/cudnn-8.0-linux-x64-v5.1/include, library: /home/amos/cudnn-8.0-linux-x64-v5.1/lib64/libcudnn.so) -- Added CUDA NVCC flags for: sm_61 -- OpenCV found (/usr/local/share/OpenCV) -- Found Atlas: /usr/include
-- Found Atlas (include: /usr/include, library: /usr/lib/libatlas.so) -- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7") -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7") -- Found NumPy: /usr/lib/python2.7/dist-packages/numpy/core/include (found suitable version "1.11.0", minimum required is "1.7.1") -- NumPy ver. 1.11.0 found (include: /usr/lib/python2.7/dist-packages/numpy/core/include) -- Boost version: 1.58.0 -- Found the following Boost libraries: -- python -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) -- Found MPI_C: /usr/local/lib/libmpi.so
-- Found MPI_CXX: /usr/local/lib/libmpi_cxx.so;/usr/local/lib/libmpi.so
-- Found Git: /usr/bin/git (found version "2.7.4") -- -- Caffe Configuration Summary -- General: -- Version : (Caffe doesn't declare its version in headers) -- Git : v0.9999-1625-gaed3884-dirty -- System : Linux -- C++ compiler : /usr/bin/c++ -- Release CXX flags : -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized -- Debug CXX flags : -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized -- Build type : Release -- -- BUILD_SHARED_LIBS : ON -- BUILD_python : ON -- BUILD_matlab : OFF -- BUILD_docs : ON -- CPU_ONLY : OFF -- -- Dependencies: -- BLAS : Yes (Atlas) -- Boost : Yes (ver. 1.58) -- glog : Yes -- gflags : Yes -- protobuf : Yes (ver. 2.6.1) -- lmdb : Yes (ver. 0.9.17) -- Snappy : Yes (ver. 1.1.3) -- LevelDB : Yes (ver. 1.18) -- OpenCV : Yes (ver. 3.1.0) -- CUDA : Yes (ver. 8.0) -- -- NVIDIA CUDA: -- Target GPU(s) : Auto -- GPU arch(s) : sm_61 -- cuDNN : Yes -- -- Python: -- Interpreter : /usr/bin/python2.7 (ver. 2.7.12) -- Libraries : /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12) -- NumPy : /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0) -- -- Documentaion: -- Doxygen : No -- config_file :
-- -- Install: -- Install path : /home/amos/person_search/caffe/build/install -- -- Configuring done -- Generating done -- Build files have been written to: /home/amos/person_search/caffe/build amos@Amos:~/person_search/caffe/build$ make -j8 && make install [ 0%] Running C++/Python protocol buffer compiler on /home/amos/person_search/caffe/src/caffe/proto/caffe.proto Scanning dependencies of target proto [ 1%] Building CXX object src/caffe/CMakeFiles/proto.dir///include/caffe/proto/caffe.pb.cc.o [ 1%] Linking CXX static library ../../lib/libproto.a [ 1%] Built target proto [ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o [ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o [ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000905_00000000-5_im2col.cpp4.ii". /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

CMake Error at cuda_compile_generated_im2col.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_im2col.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:70: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o' failed make[2]: [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o] Error 1 make[2]: Waiting for unfinished jobs.... 1 error detected in the compilation of "/tmp/tmpxft_000008fb_00000000-5_labeled_matching_layer.cpp4.ii". CMake Error at cuda_compile_generated_labeled_matching_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_labeled_matching_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:91: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000090e_00000000-5_concat_layer.cpp4.ii". CMake Error at cuda_compile_generated_concat_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_concat_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:84: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000927_00000000-5_cudnn_conv_layer.cpp4.ii". 1 error detected in the compilation of "/tmp/tmpxft_00000920_00000000-5_absval_layer.cpp4.ii". CMake Error at cuda_compile_generated_cudnn_conv_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_conv_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:98: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o] Error 1 CMake Error at cuda_compile_generated_absval_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_absval_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:77: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o] Error 1 1 error detected in the compilation of "/tmp/tmpxft_00000930_00000000-5_pooling_layer.cpp4.ii". CMake Error at cuda_compile_generated_pooling_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_pooling_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:105: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o] Error 1 /home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000919_00000000-5_math_functions.cpp4.ii". CMake Error at cuda_compile_generated_math_functions.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:63: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o' failed make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o] Error 1 1 error detected in the compilation of "/tmp/tmpxft_00000937_00000000-5_cudnn_softmax_layer.cpp4.ii". CMake Error at cuda_compile_generated_cudnn_softmax_layer.cu.o.cmake:266 (message): Error generating file /home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_softmax_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:427: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o' failed make[2]: [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o] Error 1 CMakeFiles/Makefile2:240: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed make[1]: [src/caffe/CMakeFiles/caffe.dir/all] Error 2 Makefile:127: recipe for target 'all' failed make: *** [all] Error 2

XinshaoWang commented 7 years ago

@Cysu @liuajian I could not build caffe successfully using cudnn v5.1 (I downloaded it and linked it by cmake command) or cudnn v6.0(which I used for py-faster-rcnn, combined in cuda as default cudnn version)

When I build caffe without cuDNN library support by setting USE_CUDNN OFF in the CMakeLists.txt, I build caffe successfully.

Could you please tell me how to build it with cudnn support? Thanks very much.

Cysu commented 7 years ago

Could you please check if there are any cudnn under your cuda root? For example, /usr/local/cuda/include/cudnn.hpp. If so, please remove the cudnn files (both .hpp and .so) and try it again with v5.1.

XinshaoWang commented 7 years ago

@Cysu Yes, it works now when I removed the cudnn v6.0 files from cuda root. Thanks very much.

I thought I could build with cudnn v5.1 support just by specifying the path in the cmake command. But it seems that the cudnn files in the cuda root would influencing the building.

Cysu commented 7 years ago

@XinshaoWang Great to know that!

ShuangLI59 / person_search

ERROR : Run demo.py --gpu 0 #51