search

facebookarchive / caffe2

Caffe2 is a lightweight, modular, and scalable deep learning framework.
https://caffe2.ai
Apache License 2.0
8.42k stars 1.94k forks source link

Installation issue: Unable to find Eigen #738

Open akshay-raj-dhamija opened 7 years ago

akshay-raj-dhamija commented 7 years ago

Hello,

I get the following error while trying to make from the commit acee64a9f95612e69d0a615c09e0676ac076849d. I get the same error in various earlier commits whereas the commit 0a3c8413fc26cffa255d34ba7f4830e6c9724854 is working fine. Eigen is already installed.

Building CXX object caffe2/CMakeFiles/Caffe2_CPU.dir/contrib/gloo/allreduce_ops.cc.o
In file included from /scratch/adhamija/caffe2/caffe2/core/context.h:12:0,
                 from /scratch/adhamija/caffe2/caffe2/core/tensor.h:14,
                 from /scratch/adhamija/caffe2/caffe2/core/net.h:18,
                 from /scratch/adhamija/caffe2/caffe2/core/operator.h:13,
                 from /scratch/adhamija/caffe2/caffe2/contrib/gloo/allreduce_ops.h:6,
                 from /scratch/adhamija/caffe2/caffe2/contrib/gloo/allreduce_ops.cc:1:
/scratch/adhamija/caffe2/caffe2/utils/math.h:20:22: fatal error: Eigen/Core: No such file or directory
 #include "Eigen/Core"
                      ^
compilation terminated.

Below is the complete make output

-- The BLAS backend of choice:Eigen
-- Could NOT find NNPACK (missing:  NNPACK_INCLUDE_DIR NNPACK_LIBRARY) 
-- Will try to build NNPACK from source. If anything fails, follow the NNPACK prerequisite installation steps.
CMake Warning at cmake/External/nnpack.cmake:157 (message):
  NNPACK is chosen to be installed, but confu and ninja that are needed by it
  are not installed.  As a result we won't build with NNPACK.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:52 (include)
  CMakeLists.txt:72 (include)

CMake Warning at cmake/Dependencies.cmake:57 (message):
  Not compiling with NNPACK.  Suppress this warning with -DUSE_NNPACK=OFF
Call Stack (most recent call first):
  CMakeLists.txt:72 (include)

-- Found gflags  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Found system gflags install.
-- Found glog    (include: /usr/local/include, library: /usr/local/lib/libglog.so)
-- Found system glog install.

-- Found PythonInterp: /home/adhamija/anaconda2/bin/python (found version "2.7.12") 
-- Found lmdb    (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy  (include: /usr/include, library: /usr/lib/libsnappy.so)
-- Could NOT find RocksDB (missing:  RocksDB_INCLUDE_DIR RocksDB_LIBRARIES) 
CMake Warning at cmake/Dependencies.cmake:138 (message):
  Not compiling with RocksDB.  Suppress this warning with -DUSE_ROCKSDB=OFF
Call Stack (most recent call first):
  CMakeLists.txt:72 (include)

-- OpenCV found (/home/adhamija/anaconda2/share/OpenCV)

-- Found PythonInterp: /home/adhamija/anaconda2/bin/python (found suitable version "2.7.12", minimum required is "2.7") 

-- NumPy ver. 1.12.1 found (include: /home/adhamija/anaconda2/lib/python2.7/site-packages/numpy/core/include)
-- Could NOT find pybind11 (missing:  pybind11_INCLUDE_DIR) 
-- MPI support found
-- MPI compile flags: 
-- MPI include path: /usr/lib/openmpi/include/usr/lib/openmpi/include/openmpi
-- MPI LINK flags path: 
-- MPI libraries: /usr/lib/libmpi_cxx.so/usr/lib/libmpi.so/usr/lib/x86_64-linux-gnu/libdl.so/usr/lib/x86_64-linux-gnu/libhwloc.so
CMake Warning at cmake/Dependencies.cmake:252 (message):
  OpenMPI found, but it is not built with CUDA support.
Call Stack (most recent call first):
  CMakeLists.txt:72 (include)

-- Adding -fopenmp
-- CUDA detected: 8.0
-- Added CUDA NVCC flags for: sm_52
-- Found libcuda: /usr/lib/x86_64-linux-gnu/libcuda.so
-- Found libnvrtc: /usr/local/cuda/lib64/libnvrtc.so
-- Found cuDNN: v5.1.10  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Could NOT find NCCL (missing:  NCCL_INCLUDE_DIR NCCL_LIBRARY) 
-- NCCL: /scratch/adhamija/caffe2/third_party/nccl/build/lib/libnccl_static.a
-- Could NOT find CUB (missing:  CUB_INCLUDE_DIR) 
-- Could NOT find Gloo (missing:  Gloo_INCLUDE_DIR Gloo_LIBRARY) 
-- MPI include path: /usr/lib/openmpi/include/usr/lib/openmpi/include/openmpi
-- MPI libraries: /usr/lib/libmpi_cxx.so/usr/lib/libmpi.so/usr/lib/x86_64-linux-gnu/libdl.so/usr/lib/x86_64-linux-gnu/libhwloc.so
-- CUDA detected: 8.0
-- Found libcuda: /usr/lib/x86_64-linux-gnu/libcuda.so
-- Found libnvrtc: /usr/local/cuda/lib64/libnvrtc.so
-- Does not need to define long separately.
-- This compiler has builtin_cpu_supports feature.
-- Turning off deprecation warning due to glog.
-- GCC 4.8.4: Adding gcc and gcc_s libs to link line
-- Include NCCL operators
-- Including image processing operators
-- Excluding video processing operators due to no opencv
-- Excluding mkl operators as we are not using mkl
-- Automatically generating missing __init__.py files.
-- 
-- ******** Summary ********
-- General:
--   Git version           : 
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 4.8.4
--   Protobuf compiler     : /usr/bin/protoc
--   CXX flags             :  -fopenmp -Wno-deprecated -std=c++11 -fPIC -Wno-narrowing
--   Build type            : Release
--   Compile definitions   : CAFFE2_USE_EIGEN_FOR_BLAS;CAFFE2_USE_GOOGLE_GLOG;CAFFE2_USE_GFLAGS;EIGEN_MPL2_ONLY;CAFFE2_FORCE_FALLBACK_CUDA_MPI
-- 
--   BUILD_SHARED_LIBS     : ON
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.12
--     Python library      : /usr/lib/x86_64-linux-gnu/libpython2.7.so
--   USE_CUDA              : ON
--     CUDA version        : 8.0
--   USE_CNMEM             : OFF
--   USE_NERVANA_GPU       : OFF
--   USE_GLOG              : ON
--   USE_GFLAGS            : ON
--   USE_LMDB              : ON
--     LMDB version        : 0.9.10
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.15
--     Snappy version      : 1.1.0
--   USE_OPENCV            : ON
--     OpenCV version      : 3.1.0
--   USE_FFMPEG            : 
--   USE_ZMQ               : OFF
--   USE_ROCKSDB           : OFF
--   USE_MPI               : ON
--   USE_NCCL              : ON
--   USE_NNPACK            : OFF
--   USE_OPENMP            : ON
--   USE_REDIS             : OFF
--   USE_GLOO              : ON
-- Configuring done
-- Generating done
-- Build files have been written to: /scratch/adhamija/caffe2/build
make[1]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[2]: Entering directory `/scratch/adhamija/caffe2/build'
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  2%] Built target nccl_external
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  2%] Built target gmock
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  3%] Built target gmock_main
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  3%] Built target gtest
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  4%] Built target gtest_main
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[  8%] Built target gloo
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[ 10%] Built target gloo_cuda
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[ 10%] Built target Caffe_PROTO
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[ 13%] Built target Caffe2_PROTO
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
Scanning dependencies of target Caffe2_CPU
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
make[3]: Entering directory `/scratch/adhamija/caffe2/build'
/usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
[ 13%] /usr/bin/cmake: /home/adhamija/anaconda2/lib/libcurl.so.4: no version information available (required by /usr/bin/cmake)
Building CXX object caffe2/CMakeFiles/Caffe2_CPU.dir/contrib/gloo/allreduce_ops.cc.o
In file included from /scratch/adhamija/caffe2/caffe2/core/context.h:12:0,
                 from /scratch/adhamija/caffe2/caffe2/core/tensor.h:14,
                 from /scratch/adhamija/caffe2/caffe2/core/net.h:18,
                 from /scratch/adhamija/caffe2/caffe2/core/operator.h:13,
                 from /scratch/adhamija/caffe2/caffe2/contrib/gloo/allreduce_ops.h:6,
                 from /scratch/adhamija/caffe2/caffe2/contrib/gloo/allreduce_ops.cc:1:
/scratch/adhamija/caffe2/caffe2/utils/math.h:20:22: fatal error: Eigen/Core: No such file or directory
 #include "Eigen/Core"
                      ^
compilation terminated.
make[3]: *** [caffe2/CMakeFiles/Caffe2_CPU.dir/contrib/gloo/allreduce_ops.cc.o] Error 1
make[3]: Leaving directory `/scratch/adhamija/caffe2/build'
make[2]: *** [caffe2/CMakeFiles/Caffe2_CPU.dir/all] Error 2
make[2]: Leaving directory `/scratch/adhamija/caffe2/build'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/scratch/adhamija/caffe2/build'
make: *** [all] Error 2
lukeyeager commented 7 years ago

Try reverting this change and see if the warning is actually useful for you: https://github.com/caffe2/caffe2/pull/729

akshay-raj-dhamija commented 7 years ago

I tried to change find_package(Eigen3 QUIET) in cmake to find_package(Eigen3) but the error is the same.

lukeyeager commented 7 years ago

Well that's good - at least my change didn't make anything worse!

I'd try uninstalling whatever version of Eigen you have and falling back to the version that ships with caffe2 under third_party/.

akshay-raj-dhamija commented 7 years ago

I found the error was caused in commit 63d4a406be041e04ade70e41f59a66fca197b8b2, specifically in line 203 where EIGEN3_INCLUDE_DIRS was used, where as cmake uses EIGEN3_INCLUDE_DIR

pietern commented 7 years ago

What Linux distribution are you using here? Eigen3 sets both variables, so something else must be responsible for detecting your Eigen version on your system.

gemarcano commented 7 years ago

I'm using Gentoo Linux with Eigen3 3.3.3, and it looks like I'm running into this issue as well. Looking at the CMake output, my system Eigen is being detected, but here are the relevant lines from the output:

-- Found Eigen3: /usr/include/eigen3 (Required is at least version "2.91.0") 
-- Found system Eigen at

For some reason EIGEN3_INCLUDE_DIRS is not set. I'm trying to track down why, since this variable should be set by Eigen3Config.cmake.

gemarcano commented 7 years ago

I'm attaching the cmake --trace ../ output from the build directory: cmake.log

The major finding here is that Eigen3's Config.cmake doesn't seem to be picked up. This may be a problem with Eigen3 (I remember seeing a Debian patch for this when looking through Ubuntu's files to see what differs between Gentoo and Ubuntu).

gemarcano commented 7 years ago

Alright, I think I tracked down what's going on. I upgraded my Eigen3 installation to 3.3.4 just in case.

Based on what I can, FindEigen3.cmake sets the version number to 2.91.0 at first. Later, it invokes find_package(Eigen3 2.91.0 NO_MODULE QUIET ) which jumps to Eigen3ConfigVersion.cmake. There, PACKAGE_VERSION_COMPATIBLE is set to false:

/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(12):  set(PACKAGE_VERSION 3.3.4 )
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(14):  if(PACKAGE_VERSION VERSION_LESS PACKAGE_FIND_VERSION )
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(16):  else()
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(18):  if(3.3.4 MATCHES ^([0-9]+)\. )
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(19):  set(CVF_VERSION_MAJOR 3 )
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(24):  if(PACKAGE_FIND_VERSION_MAJOR STREQUAL CVF_VERSION_MAJOR )
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(26):  else()
/usr/share/eigen3/cmake/Eigen3ConfigVersion.cmake(27):  set(PACKAGE_VERSION_COMPATIBLE FALSE )

Now, I guess that because PACKAGE_VERSION_COMPATIBLE is false, cmake refuses to process Eigen3Config.cmake. In fact, hacking that check above to return true, causes Eigen3Config.cmake to be processed and the DIRS variable is set properly.

One way to fix this is to have caffe explicitly demand for an Eigen3 version that's compatible, well, with Eigen3 (I'm guessing that's 3.0.0+), instead of relying on the default check of 2.91.0.

Changing the find_package call in cmake/Dependencies.cmake for Eigen3 to find_package(Eigen3 3.0.0) makes this work on my system.

pietern commented 6 years ago

@gemarcano Which package installed FindEigen3.cmake for you? I tried repro'ing but see that Eigen 3 provided its own FindEigen3.cmake which was up to date with its version. If your FindEigen3.cmake expands the version to 2.9.1 then I suppose it's pretty old and not provided by the Eigen package itself.

ducha-aiki commented 6 years ago

@gemarcano I had the same issue (installed earlier Eigen in system) and confirm that "Changing the find_package call in cmake/Dependencies.cmake for Eigen3 to find_package(Eigen3 3.0.0) makes this work on my system." helps