Caffe2 install requires build directory?

[orionr]

This issue was originally submitted as a Caffe2 issue at https://github.com/caffe2/caffe2/issues/1999

I think Detectron requires Caffe2 to be built with

mkdir build && cd build
cmake ..
make
export LD_LIBRARY_PATH="${caffe2_dir}/build/lib:${LD_LIBRARY_PATH}"
export PYTHONPATH="${caffe2_dir}/build:${PYTHONPATH}"

per https://github.com/facebookresearch/Detectron/blob/master/lib/utils/env.py#L80

However, we want to support people using conda and virtualenv.

Just putting this up to flag it.

[orionr]

Also FYI that our recommended install for Caffe2 (on Ubuntu, for example) is now conda prebuilt binaries at

https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=prebuilt

[ir413]

Hi @orionr - yes, when searching for Caffe2 detectron ops library we used to assume that Caffe2 build dir is in python path. We recently improved this in 92f6cd6 and having the Caffe2 build directory is not required anymore.

If rebasing on 92f6cd6 does not solve the issue, let me know where the Caffe2 detectron ops lib is placed in the conda/virtualenv installations and we can make sure that Detectron is able to find it.

Note that the code you linked to is used for finding custom Detectron ops (from detectron/lib/ops) and not the Caffe2 detectron ops (which are handled by get_detectron_ops_lib).

[Dimlight]

@ir413 can you please help me ? I also posted at detectron issues but nobody replied.

I installed caffe2 using this command :

./scripts/build_anaconda.sh --skip-tests --cuda 8.0.62 --cudnn 6.0.21

I could successfully build.

CAFFE2_CMAKE_ARGS='-DUSE_LEVELDB=OFF -DUSE_LMDB=OFF -DUSE_CUDA=ON -DUSE_NCCL=ON'

environment location: /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl

"Copying /home/ekram/pytorch to /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/work source tree in: /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/work

• echo 'Installing caffe2 to /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl' Installing caffe2 to /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl "

-- -- **** Summary **** -- General: -- CMake version : 3.11.1 -- CMake command : /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/bin/cmake -- Git version : v0.1.11-8310-ga257bd1 -- System : Linux -- C++ compiler : /usr/bin/c++ -- C++ compiler version : 5.4.0 -- BLAS : Eigen -- CXX flags : -fvisibility-inlines-hidden -DONNX_NAMESPACE=onnx_c2 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization -- Build type : Release -- Compile definitions : -- -- BUILD_CAFFE2 : ON -- BUILD_ATEN : OFF -- BUILD_BINARY : ON -- BUILD_CUSTOM_PROTOBUF : ON -- Link local protobuf : ON -- BUILD_DOCS : OFF -- BUILD_PYTHON : ON -- Python version : 2.7.15 -- Python includes : /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/python2.7 -- BUILD_SHARED_LIBS : ON -- BUILD_TEST : ON -- USE_ASAN : OFF -- USE_CUDA : ON -- CUDA static link : OFF -- USE_CUDNN : ON -- CUDA version : 7.5 -- cuDNN version : 6.0.21 -- CUDA root directory : /usr -- CUDA library : /usr/lib/x86_64-linux-gnu/libcuda.so -- cudart library : /usr/lib/x86_64-linux-gnu/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so -- cublas library : /usr/lib/x86_64-linux-gnu/libcublas.so;/usr/lib/x86_64-linux-gnu/libcublas_device.a -- curand library : /usr/lib/x86_64-linux-gnu/libcurand.so -- CuDNN library : /usr/local/cuda/lib64/libcudnn.so -- nvrtc : /usr/lib/x86_64-linux-gnu/libnvrtc.so -- CUDA include path : /usr/include -- NVCC executable : /usr/bin/nvcc -- CUDA host compiler : /usr/bin/cc -- USE_TENSORRT : OFF -- USE_EIGEN_FOR_BLAS : ON -- USE_FFMPEG : OFF -- USE_GFLAGS : ON -- USE_GLOG : ON -- USE_GLOO : ON -- USE_GLOO_IBVERBS : OFF -- USE_LEVELDB : OFF -- USE_LITE_PROTO : OFF -- USE_LMDB : OFF -- USE_METAL : OFF -- USE_MKL : -- USE_MOBILE_OPENGL : OFF -- USE_MPI : ON -- USE_NCCL : ON -- USE_SYSTEM_NCCL : OFF -- USE_NERVANA_GPU : OFF -- USE_NNPACK : ON -- USE_OBSERVERS : ON -- USE_OPENCL : OFF -- USE_OPENCV : ON -- OpenCV version : 3.3.1 -- USE_OPENMP : OFF -- USE_PROF : OFF -- USE_REDIS : OFF -- USE_ROCKSDB : OFF -- USE_ZMQ : OFF -- Configuring done -- Generating done -- Build files have been written to: /home/ekram/anaconda2/conda-bld/caffe2-cuda8.0.61-cudnn6.0.21_1526196669948/work/build

cd ~ && python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"

output : Success

But still when i try to import import caffe2.python.core and tried to run this command to test my GPU support, still i got those warnings.

cd /detectron/detectron/tests python test_spatial_narrow_as_op.py

output : WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode. WARNING:root:Debug message: No module named caffe2_pybind11_state_gpu E0513 12:29:40.132611 3529 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E0513 12:29:40.132633 3529 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E0513 12:29:40.132638 3529 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. Found Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so Traceback (most recent call last): File "test_spatial_narrow_as_op.py", line 88, in c2_utils.import_detectron_ops() File "/home/ekram/detectron/detectron/utils/c2.py", line 42, in import_detectron_ops dyndep.InitOpsLibrary(detectron_ops_lib) File "/home/ekram/anaconda2/lib/python2.7/site-packages/caffe2/python/dyndep.py", line 35, in InitOpsLibrary _init_impl(name) File "/home/ekram/anaconda2/lib/python2.7/site-packages/caffe2/python/dyndep.py", line 48, in _init_impl ctypes.CDLL(path) File "/home/ekram/anaconda2/lib/python2.7/ctypes/init.py", line 366, in init self._handle = _dlopen(self._name, mode) OSError: /usr/local/lib/libcaffe2_gpu.so: undefined symbol: _ZN2cv12_OutputArrayC1ERNS_3MatE

Operating system: Linux 16.04
Compiler version:
CUDA version: 8.0.61
cuDNN version: 6.0.21
NVIDIA driver version: 384.130
GPU models (for all devices if they are not all the same): GTX 1050ti
PYTHONPATH environment variable: confused about this. 
python --version output: Python 2.7.14 :: Anaconda, Inc.
Anything else that seems relevant: