Open II-Matto opened 6 years ago
gadcam
commented
6 years ago To me, the interesting part is here
CMake Warning at /home/
/lib/cmake-3.10.2-Linux-x86_64/share/cmake-3.10/Modules/FindProtobuf.cmake:455 (message): Protobuf compiler version 3.2.0 doesn't match library version 2.5.0
Can you check if this solution is working for you ? https://github.com/caffe2/caffe2/issues/1684#issuecomment-356028126
II-Matto
commented
6 years ago @gadcam Hi, the protobuf is OK for me. The version of the protobuf used in my Anaconda and in building Caffe2 are both 3.5.0, thus being consistent. With inconsistent versions of protobuf, I will get the following error when importing caffe2_pb:
__init__() got an unexpected keyword argument 'file'What is weird is that when I upgraded my protobuf in Anaconda using conda install <download_tar_file>, it still resulted in errors for importing caffe2_pb. The error was resolved after I tried again using pip install <download_whl_file>.
My problem in make ops is now solved by modifying the CMakeLists.txt in the Detectron root. I changed the following line:
list(APPEND CUDA_INCLUDE_DIRS -I${CAFFE2_INCLUDE_DIRS})-I.${CUDNN_INCLUDE_DIR}.By message out the CUDA_INCLUDE_DIRS, I found that with -I one would get something like:
"<dir_1>:-I<dir_2>"I guess this actually results in invalid include directory configuration. Since my problem is clearly caused by not finding the cuDNN header file, I further add the CUDNN_INCLUDE_DIR which is passed in with the cmake command.
With the cuDNN header file problem solved, I met another problem of not finding -lcaffe2:cudnn as described in issue https://github.com/facebookresearch/Detectron/issues/456. The solution is proposed to simply remove this option in the corresponding file, and it worked for me. Besides, I also tried changing -lcaffe2:cudnn into -lcaffe2_gpu -L<path_to_caffe2_install_dir>/lib, which also solves the problem. I am not sure if it is necessary to link the caffe2_gpu library.
Now I can successfully make ops and obtain the library files, i.e. libcaffe2_detectron_custom_ops_gpu.so and libcaffe2_detectron_custom_ops.so. I tried loading them with the following code:
from caffe2.python import dyndep, workspace
# Load only one of the following two libraries, or it may report "Offending key".
# dyndep.InitOpsLibrary('/path/to/libcaffe2_detectron_custom_ops.so')
dyndep.InitOpsLibrary('/path/to/libcaffe2_detectron_custom_ops_gpu.so')
print 'ZeroEven' in workspace.RegisteredOperators()The output is True. This is just a simple test. Maybe more tests are needed to see if this operator can work as expected.
II-Matto
commented
6 years ago It seems that the -lcaffe2::cudnn problem is due to the Caffe2 target. The caffe2::cudnn target is defined in cmake/public/cuda.cmake, added in cmake/Dependencies.cmake, and "linked" in caffe2/CMakeLists.txt.
So I tried adding the following code (i.e. definition of caffe2::cudnn target) in Detectron CMakeLists.txt when ${CAFFE2_FOUND_CUDA} is true.
# cudnn
add_library(caffe2::cudnn UNKNOWN IMPORTED)
set_property(
TARGET caffe2::cudnn PROPERTY IMPORTED_LOCATION
${CUDNN_LIBRARY})
set_property(
TARGET caffe2::cudnn PROPERTY INTERFACE_INCLUDE_DIRECTORIES
${CUDNN_INCLUDE_DIR})Then my cmake and make commands work without errors.
Caffe2_DIR=/caffe2/install/dir cmake -DCUDNN_INCLUDE_DIR=/cudnn/include/dir -DCUDNN_LIBRARY=/cudnn/lib/file.so ..
makeBTW, I also add cmake_policy(SET CMP0028 NEW) in CMakeLists.txt to suppress some warnings.
According to the comments in Detectron CMakeLists.txt. The additional -I prefix in list(APPEND CUDA_INCLUDE_DIRS -I<dir_1> -I<dir_2>) is required for CMake of versions < 3.7 (my CMake version is 3.10.2). So I guess a better solution is to check the CMake version and add the -I prefix when necessary.
Hi, @ir413. The Detectron CMakeLists.txt probably needs some fix.
ir413
commented
6 years ago Thanks for the provided information. Good point about CMake versions. Will look into this.
Expected results
After successfully building Caffe2, I would like to compile the custom operators in Detectron. So I tried the
make ops.FYI, I have already passed the following tests:
importtests.pytorch/build/caffe2/python/operator_test/relu_op_test.pyDetectron/detectron/tests/test_spatial_narrow_as_op.pyActual results
It gave the following errors, reporting that the
cudnn.hcould not be found. Though the problem seems to be very obvious, but some general solutions just did not work. I have tried adding-I<cudnn_include_dir>in various places and settingCPPFLAGS, but none of them helps.Detailed steps to reproduce
Instead of directly running
make opsin Detectron root directory, I ran the following command so that the cuDNN path can be set manually.System information
PYTHONPATHenvironment variable: (pytorch/builddirectory)python --versionoutput: Python 2.7.13 :: Anaconda custom (64-bit)The CMake outputs are given below.