search

facebookarchive / caffe2

Caffe2 is a lightweight, modular, and scalable deep learning framework.
https://caffe2.ai
Apache License 2.0
8.42k stars 1.94k forks source link

Build from source with anaconda with python2.7 #2360

Closed KMnP closed 6 years ago

KMnP commented 6 years ago

Hi, I follows guild to build with anaconda, with following command for GPU (in python 2.7 environment): CONDA_INSTALL_LOCALLY=1 BUILD_ENVIRONMENT=conda-cuda-macos ./scripts/build_anaconda.sh

But it seems like only supported for python 3.6? is there any way to install it with python 2? Thanks!

System information

**** Summary ****

UnsatisfiableError: The following specifications were found to be in conflict:

KMnP commented 6 years ago

Here is my output of cmake command:

`-- The CXX compiler identification is GNU 5.4.0 -- The C compiler identification is GNU 5.4.0 -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Performing Test CAFFE2_LONG_IS_INT32_OR_64 -- Performing Test CAFFE2_LONG_IS_INT32_OR_64 - Success -- Does not need to define long separately. -- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED -- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success -- std::exception_ptr is supported. -- Performing Test CAFFE2_IS_NUMA_AVAILABLE -- Performing Test CAFFE2_IS_NUMA_AVAILABLE - Failed -- NUMA is not available -- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING -- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed -- Turning off deprecation warning due to glog. -- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS -- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success -- Current compiler supports avx2 extention. Will build perfkernels. -- Build type not set - defaulting to Release -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE
-- Caffe2: Found protobuf with old-style protobuf targets. -- Caffe2 protobuf include directory: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -- Found Git: /usr/bin/git (found version "2.7.4") -- The BLAS backend of choice:Eigen -- Could NOT find NNPACK (missing: NNPACK_INCLUDE_DIR NNPACK_LIBRARY PTHREADPOOL_LIBRARY CPUINFO_LIBRARY) -- Brace yourself, we are building NNPACK -- The ASM compiler identification is GNU -- Found assembler: /usr/bin/cc -- Found PythonInterp: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/python (found version "3.6.4") -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - yes -- Caffe2: Found gflags with new-style gflags target. -- Caffe2: Cannot find glog automatically. Using legacy find. -- Found glog: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-- Caffe2: Found glog (include: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include, library: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/libglog.so) -- git Version: v0.0.0-dirty -- Version: 0.0.0 -- Performing Test HAVE_CXX_FLAG_STD_CXX11 -- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success -- Performing Test HAVE_CXX_FLAG_WALL -- Performing Test HAVE_CXX_FLAG_WALL - Success -- Performing Test HAVE_CXX_FLAG_WEXTRA -- Performing Test HAVE_CXX_FLAG_WEXTRA - Success -- Performing Test HAVE_CXX_FLAG_WSHADOW -- Performing Test HAVE_CXX_FLAG_WSHADOW - Success -- Performing Test HAVE_CXX_FLAG_WERROR -- Performing Test HAVE_CXX_FLAG_WERROR - Success -- Performing Test HAVE_CXX_FLAG_PEDANTIC -- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success -- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS -- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success -- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 -- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed -- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL -- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success -- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING -- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success -- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT -- Performing Test HAVE_CXX_FLAG_WZERO_AS_NULL_POINTER_CONSTANT - Success -- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING -- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success -- Performing Test HAVE_CXX_FLAG_WD654 -- Performing Test HAVE_CXX_FLAG_WD654 - Failed -- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY -- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed -- Performing Test HAVE_CXX_FLAG_COVERAGE -- Performing Test HAVE_CXX_FLAG_COVERAGE - Success -- Performing Test HAVE_STD_REGEX -- Performing Test HAVE_STD_REGEX -- Performing Test HAVE_STD_REGEX -- success -- Performing Test HAVE_GNU_POSIX_REGEX -- Performing Test HAVE_GNU_POSIX_REGEX -- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile -- Performing Test HAVE_POSIX_REGEX -- Performing Test HAVE_POSIX_REGEX -- Performing Test HAVE_POSIX_REGEX -- success -- Performing Test HAVE_STEADY_CLOCK -- Performing Test HAVE_STEADY_CLOCK -- Performing Test HAVE_STEADY_CLOCK -- success -- Found LMDB: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-- Found lmdb (include: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include, library: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/liblmdb.so) -- Found LevelDB: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-- Found LevelDB (include: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include, library: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/libleveldb.so) -- Found Snappy: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include
-- Found Snappy (include: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include, library: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/libsnappy.so) -- Could NOT find Numa (missing: Numa_INCLUDE_DIR Numa_LIBRARIES) CMake Warning at cmake/Dependencies.cmake:208 (message): Not compiling with NUMA. Suppress this warning with -DUSE_NUMA=OFF Call Stack (most recent call first): CMakeLists.txt:103 (include)

-- OpenCV found (/home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/share/OpenCV) CMake Warning at cmake/Dependencies.cmake:273 (find_package): By not providing "FindEigen3.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "Eigen3", but CMake did not find one.

Could not find a package configuration file provided by "Eigen3" with any of the following names:

Eigen3Config.cmake
eigen3-config.cmake

Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set "Eigen3_DIR" to a directory containing one of the above files. If "Eigen3" provides a separate development package or SDK, be sure it has been installed. Call Stack (most recent call first): CMakeLists.txt:103 (include)

-- Did not find system Eigen. Using third party subdirectory. -- Found PythonInterp: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/python (found suitable version "3.6.4", minimum required is "2.7") -- Found PythonLibs: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/libpython3.6m.so (found suitable version "3.6.4", minimum required is "2.7") -- Found NumPy: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/numpy/core/include (found version "1.12.1") -- NumPy ver. 1.12.1 found (include: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/python3.6/site-packages/numpy/core/include) -- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR) -- Found CUDA: /usr/local/cuda-8.0 (found suitable version "8.0", minimum required is "7.0") -- Found CUDNN: /usr/include
-- Caffe2: CUDA detected: 8.0 -- Found cuDNN: v7.0.5 (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so) -- Automatic GPU detection returned 6.1 6.1. -- Added CUDA NVCC flags for: sm_61 -- Could NOT find NCCL (missing: NCCL_INCLUDE_DIRS NCCL_LIBRARIES) -- Could NOT find CUB (missing: CUB_INCLUDE_DIR) -- Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY) -- CUDA detected: 8.0 -- Found libcuda: /usr/lib/x86_64-linux-gnu/libcuda.so -- Found libnvrtc: /usr/lib/x86_64-linux-gnu/libnvrtc.so -- Found nccl: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/work/third_party/nccl/build/include
CMake Warning at cmake/Dependencies.cmake:460 (message): mobile opengl is only used in android or ios builds. Call Stack (most recent call first): CMakeLists.txt:103 (include)

CMake Warning at cmake/Dependencies.cmake:536 (message): Metal is only used in ios builds. Call Stack (most recent call first): CMakeLists.txt:103 (include)

-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR) -- GCC 5.4.0: Adding gcc and gcc_s libs to link line -- Include NCCL operators -- Including image processing operators -- Excluding video processing operators due to no opencv -- Excluding mkl operators as we are not using mkl -- MPI operators skipped due to no MPI support -- Include Observer library -- Using lib/python3.6/site-packages as python relative installation path -- Automatically generating missing init.py files. -- -- **** Summary **** -- General: -- CMake version : 3.9.4 -- CMake command : /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/cmake -- Git version : v0.8.1-1385-gb3046c4-dirty -- System : Linux -- C++ compiler : /usr/bin/c++ -- C++ compiler version : 5.4.0 -- Protobuf compiler : /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/bin/protoc -- Protobuf include path : /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include -- Protobuf libraries : /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/lib/libprotobuf.so;-lpthread -- BLAS : Eigen -- CXX flags : -Wno-deprecated -DONNX_NAMESPACE=onnx_c2 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization -- Build type : Release -- Compile definitions : -- -- BUILD_BINARY : ON -- BUILD_CUSTOM_PROTOBUF : OFF -- BUILD_DOCS : OFF -- BUILD_PYTHON : ON -- Python version : 3.6.4 -- Python includes : /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place/include/python3.6m -- BUILD_SHARED_LIBS : ON -- BUILD_TEST : ON -- USE_ATEN : OFF -- USE_ASAN : OFF -- USE_CUDA : ON -- CUDA version : 8.0 -- CuDNN version : 7.0.5 -- CUDA root directory : /usr/local/cuda-8.0 -- CUDA library : /usr/lib/x86_64-linux-gnu/libcuda.so -- CUDA NVRTC library : /usr/lib/x86_64-linux-gnu/libnvrtc.so -- CUDA runtime library: /usr/local/cuda-8.0/lib64/libcudart.so -- CUDA include path : /usr/local/cuda-8.0/include -- NVCC executable : /usr/local/cuda-8.0/bin/nvcc -- CUDA host compiler : /usr/bin/cc -- USE_EIGEN_FOR_BLAS : 1 -- USE_FFMPEG : OFF -- USE_GFLAGS : ON -- USE_GLOG : ON -- USE_GLOO : ON -- USE_LEVELDB : ON -- LevelDB version : 1.20 -- Snappy version : .. -- USE_LITE_PROTO : OFF -- USE_LMDB : ON -- LMDB version : 0.9.21 -- USE_METAL : OFF -- USE_MKL : -- USE_MOBILE_OPENGL : OFF -- USE_MPI : OFF -- USE_NCCL : ON -- USE_NERVANA_GPU : OFF -- USE_NNPACK : ON -- USE_OBSERVERS : ON -- USE_OPENCV : ON -- OpenCV version : 3.1.0 -- USE_OPENMP : OFF -- USE_PROF : OFF -- USE_REDIS : OFF -- USE_ROCKSDB : OFF -- USE_ZMQ : OFF -- Configuring done -- Generating done -- Build files have been written to: /home/user/anaconda3/conda-bld/caffe2-cuda-cudnn_1521670152762/work/build`

pjh5 commented 6 years ago

This script uses whatever python you have for your default. If python version returns python 2 then it will build for python 2. If you have python 2 but want to build for python 3, then create a new conda environment for python 3 with conda create -yn caffe2_env --python=3

KMnP commented 6 years ago

My default is python3, but wanna install caffe2 at python2, so I activated python2.7 before installing caffe2. But it seems like some dependencies are built for python3 only. Is there any way to solve this problem?

(I installed caffe2 with default python3 last night, everything works fine.)

pjh5 commented 6 years ago

@KMnP conda will pick which packages to use depending on what python version you are using when you run the conda-build commands. What is output when you run python --version ?

KMnP commented 6 years ago

Thank you! I tried to create a new conda environment by: conda create -yn detectron_caffe2_env python=2.7

Then install caffe2 according to the Anaconda custom build, everything works sorts of fine. If I cd caffe2 folder, two tests are all fine. But if I use $HOME, then I got the error message:

(detectron_caffe2_env) user@rock-n-roll:~$ python2 -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure" Failure (detectron_caffe2_env) user@rock-n-roll:~$ python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())' WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode. WARNING:root:Debug message: liblmdb.so: cannot open shared object file: No such file or directory CRITICAL:root:Cannot load caffe2.python. Error: liblmdb.so: cannot open shared object file: No such file or directory

Everything is fine at caffe2 directory: (detectron_caffe2_env) user@rock-n-roll:~$ cd caffe2 (detectron_caffe2_env) user@rock-n-roll:~/caffe2$ python2 -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure" Success (detectron_caffe2_env) user@rock-n-roll:~/caffe2$ python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())' 2

Do you think it's a python path problem?

Here is the output: (detectron_caffe2_env) user@rock-n-roll:~$ which python /home/user/anaconda3/bin/python (detectron_caffe2_env) user@rock-n-roll:~$ python --version Python 3.6.4 :: Anaconda custom (64-bit) (detectron_caffe2_env) user@rock-n-roll:~$ echo $PYTHONPATH /home/user/anaconda3/envs/detectron_caffe2_env/lib/python2.7/site-packages/caffe2/python/build (detectron_caffe2_env) user@rock-n-roll:~$ echo $PATH /usr/local/cuda-8.0/bin:/home/user/anaconda3/bin:/home/user/anaconda3/envs/detectron_caffe2_env/bin:/usr/local/cuda-8.0/bin:/home/user/anaconda3/bin:/home/user/bin:/home/user/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin (detectron_caffe2_env) user@rock-n-roll:~$ echo $LD_LIBRARY_PATH /usr/local/cuda-8.0/lib64:/usr/local/lib:/usr/local/cuda-8.0/lib64:/usr/local/lib:

Thank you!

pjh5 commented 6 years ago

Something is wrong with your Anaconda setup. If the environment was created correctly then which python would be /home/user/anaconda3/envs/detectron_caffe2_env/bin/python and python --version would be Python 2.7 Are you sure you're activating the conda environment after you create it?

Your PATH looks wrong. You shouldn't have both /home/user/anaconda3/bin and /home/user/anaconda3/envs/detectron_caffe2_env/bin in your path, because the first will always shadow the second. You should read this too https://caffe2.ai/docs/faq.html#why-do-i-get-import-errors-in-python-when-i-try-to-use-caffe2 .

KMnP commented 6 years ago

HI Paul, I delete the path and change PYTHONPATH to the correct caffe2 build directory, somehow the python version is not an issue, I use python2 somecommand..., everything works fine now!

Thanks for the help!