rioyokotalab / caffe2

Caffe2 is a lightweight, modular, and scalable deep learning framework.
https://caffe2.ai
Other
2 stars 0 forks source link

[Warsaw] Caffe2 build Problem "cannot find -lpthreads" #18

Open Hiroki11x opened 6 years ago

Hiroki11x commented 6 years ago
pip install numpy
pip install future
pip install protobuf

CMAKE_PREFIX_PATH=/home/hiroki11x/env/local/opencv-2.4.13:/home/hiroki11x/env/local/snappy-1.1.4 cmake .. \
-DBLAS=Eigen \
-DUSE_CUDA=ON \
-DUSE_ROCKSDB=OFF \
-DUSE_GLOO=ON \
-DUSE_REDIS=OFF \
-DUSE_OPENCV=ON \
-DUSE_GFLAGS=OFF \
-DUSE_MPI=OFF \
-DCUDNN_INCLUDE_DIR=/home/hiroki11x/env/local/cudnn7/cuda/include \
-DCUDNN_LIBRARY=/home/hiroki11x/env/local/cudnn7/cuda/lib/libcudnn.so \
-DCMAKE_INSTALL_PREFIX=/home/hiroki11x/dl/caffe2/local \
..

if you want to use mpi, append following options

-DMPI_C_COMPILER=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/mic/bin/mpicc \
-DMPI_CXX_COMPILER=/opt/intel/compilers_and_libraries_2017.1.132/linux/mpi/mic/bin/mpicxx \

then , I execute cmake

Run Build Command:"/usr/bin/gmake" "cmTC_8b05e/fast"
/usr/bin/gmake -f CMakeFiles/cmTC_8b05e.dir/build.make CMakeFiles/cmTC_8b05e.dir/build
gmake[1]: Entering directory `/home/hiroki11x/dl/caffe2/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_8b05e.dir/CheckFunctionExists.c.o
/usr/bin/cc    -DCHECK_FUNCTION_EXISTS=pthread_create   -o CMakeFiles/cmTC_8b05e.dir/CheckFunctionExists.c.o   -c /home/hiroki11x/env/src/cmake-3.4.0-rc3/Modules/CheckFunctionExists.c
Linking C executable cmTC_8b05e
/home/hiroki11x/env/src/cmake-3.4.0-rc3/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8b05e.dir/link.txt --verbose=1
/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=pthread_create    CMakeFiles/cmTC_8b05e.dir/CheckFunctionExists.c.o  -o cmTC_8b05e -rdynamic -lpthreads 
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
gmake[1]: *** [cmTC_8b05e] Error 1
gmake[1]: Leaving directory `/home/hiroki11x/dl/caffe2/build/CMakeFiles/CMakeTmp'
gmake: *** [cmTC_8b05e/fast] Error 2
Hiroki11x commented 6 years ago
-- The CXX compiler identification is GNU 4.8.5

-- The C compiler identification is GNU 4.8.5
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Build type not set - defaulting to Release
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Could NOT find Protobuf (missing:  Protobuf_INCLUDE_DIR) 
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") 
-- Using protobuf compiler /home/hiroki11x/env/local/protobuf-3.3.0/bin/protoc.
-- The BLAS backend of choice:Eigen
-- Could NOT find NNPACK (missing:  NNPACK_INCLUDE_DIR NNPACK_LIBRARY PTHREADPOOL_LIBRARY) 
-- Will try to build NNPACK from source. If anything fails, follow the NNPACK prerequisite installation steps.
CMake Warning at cmake/External/nnpack.cmake:154 (message):
  NNPACK is chosen to be installed, but confu and ninja that are needed by it
  are not installed.  As a result we won't build with NNPACK.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:53 (include)
  CMakeLists.txt:73 (include)

CMake Warning at cmake/Dependencies.cmake:58 (message):
  Not compiling with NNPACK.  Suppress this warning with -DUSE_NNPACK=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)

-- Could NOT find GFlags (missing:  GFLAGS_INCLUDE_DIR GFLAGS_LIBRARY) 
CMake Warning at cmake/External/gflags.cmake:10 (message):
  gflags is not found.  Caffe2 will build without gflags support but it is
  strongly recommended that you install gflags.
Call Stack (most recent call first):
  cmake/External/glog.cmake:2 (include)
  cmake/Dependencies.cmake:65 (include)
  CMakeLists.txt:73 (include)

-- Could NOT find Glog (missing:  GLOG_INCLUDE_DIR GLOG_LIBRARY) 
CMake Warning at cmake/External/glog.cmake:13 (message):
  glog is not found.  Caffe2 will build without glog support but it is
  strongly recommended that you install glog.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:65 (include)
  CMakeLists.txt:73 (include)

CMake Warning at cmake/Dependencies.cmake:71 (message):
  Not compiling with glog.  Suppress this warning with -DUSE_GLOG=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)

-- Found PythonInterp: /home/hiroki11x/.pyenv/shims/python (found version "2.7.10") 
-- Could NOT find Benchmark (missing:  Benchmark_INCLUDE_DIR Benchmark_LIBRARY) 
CMake Error at cmake/Dependencies.cmake:99 (add_subdirectory):
  The source directory

    /home/hiroki11x/dl/caffe2/third_party/benchmark

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)

-- Could NOT find LMDB (missing:  LMDB_INCLUDE_DIR LMDB_LIBRARIES) 
CMake Warning at cmake/Dependencies.cmake:111 (message):
  Not compiling with LMDB.  Suppress this warning with -DUSE_LMDB=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)

-- Could NOT find LevelDB (missing:  LevelDB_INCLUDE LevelDB_LIBRARY) 
-- Found Snappy: /home/hiroki11x/env/local/snappy-1.1.4/include  
-- Found Snappy  (include: /home/hiroki11x/env/local/snappy-1.1.4/include, library: /home/hiroki11x/env/local/snappy-1.1.4/lib/libsnappy.so)
CMake Warning at cmake/Dependencies.cmake:127 (message):
  Not compiling with LevelDB.  Suppress this warning with -DUSE_LEVELDB=OFF
Call Stack (most recent call first):
  CMakeLists.txt:73 (include)

-- OpenCV found (/home/hiroki11x/env/local/opencv-2.4.13/share/OpenCV)
-- Found PythonInterp: /home/hiroki11x/.pyenv/shims/python (found suitable version "2.7.10", minimum required is "2.7") 
-- Found PythonLibs: /usr/lib64/libpython2.7.so (found suitable version "2.7.5", minimum required is "2.7") 
-- Found NumPy: /home/hiroki11x/.pyenv/versions/2.7.10/lib/python2.7/site-packages/numpy/core/include (found version "1.13.1") 
-- NumPy ver. 1.13.1 found (include: /home/hiroki11x/.pyenv/versions/2.7.10/lib/python2.7/site-packages/numpy/core/include)
-- Could NOT find pybind11 (missing:  pybind11_INCLUDE_DIR) 
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp  
-- Adding -fopenmp
-- CUDA detected: 8.0
-- Added CUDA NVCC flags for: sm_60
-- Found libcuda: /usr/lib64/libcuda.so
-- Found libnvrtc: /usr/local/cuda/lib64/libnvrtc.so
-- Found CUDNN: /home/hiroki11x/env/local/cudnn7/cuda/include  
-- Found cuDNN: v7.0.1  (include: /home/hiroki11x/env/local/cudnn7/cuda/include, library: /home/hiroki11x/env/local/cudnn7/cuda/lib/libcudnn.so)
-- Could NOT find NCCL (missing:  NCCL_INCLUDE_DIR NCCL_LIBRARY) 
-- NCCL: /home/hiroki11x/dl/caffe2/third_party/nccl/build/lib/libnccl_static.a
-- Could NOT find CUB (missing:  CUB_INCLUDE_DIR) 
-- Could NOT find Gloo (missing:  Gloo_INCLUDE_DIR Gloo_LIBRARY) 
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "7.0") 
-- CUDA detected: 8.0
-- Found libcuda: /usr/lib64/libcuda.so
-- Found libnvrtc: /usr/local/cuda/lib64/libnvrtc.so
-- Performing Test CAFFE2_LONG_IS_INT32_OR_64
-- Performing Test CAFFE2_LONG_IS_INT32_OR_64 - Success
-- Does not need to define long separately.
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Success
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extention. Will build perfkernels.
-- GCC 4.8.5: Adding gcc and gcc_s libs to link line
-- Include NCCL operators
-- Including image processing operators
-- Excluding video processing operators due to no opencv
-- Excluding mkl operators as we are not using mkl
-- MPI operators skipped due to no MPI support
-- Automatically generating missing __init__.py files.
-- 
-- ******** Summary ********
-- General:
--   Git version           : 
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 4.8.5
--   Protobuf compiler     : /home/hiroki11x/env/local/protobuf-3.3.0/bin/protoc
--   CXX flags             :  -fopenmp -std=c++11 -O2 -fPIC -Wno-narrowing
--   Build type            : Release
--   Compile definitions   : CAFFE2_USE_EIGEN_FOR_BLAS;EIGEN_MPL2_ONLY;CAFFE2_PERF_WITH_AVX;CAFFE2_PERF_WITH_AVX2
-- 
--   BUILD_SHARED_LIBS     : ON
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.5
--     Python library      : /usr/lib64/libpython2.7.so
--   BUILD_TEST            : ON
--   USE_CUDA              : ON
--     CUDA version        : 8.0
--   USE_CNMEM             : OFF
--   USE_NERVANA_GPU       : OFF
--   USE_GLOG              : OFF
--   USE_GFLAGS            : OFF
--   USE_LMDB              : OFF
--   USE_LEVELDB           : OFF
--   USE_OPENCV            : ON
--     OpenCV version      : 2.4.13
--   USE_FFMPEG            : 
--   USE_ZMQ               : OFF
--   USE_ROCKSDB           : OFF
--   USE_MPI               : OFF
--   USE_NCCL              : ON
--   USE_NNPACK            : OFF
--   USE_OPENMP            : ON
--   USE_REDIS             : OFF
--   USE_GLOO              : ON
-- Configuring incomplete, errors occurred!
See also "/home/hiroki11x/dl/caffe2/build/CMakeFiles/CMakeOutput.log".
See also "/home/hiroki11x/dl/caffe2/build/CMakeFiles/CMakeError.log".
Hiroki11x commented 6 years ago

Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR) Could NOT find NNPACK (missing: NNPACK_INCLUDE_DIR NNPACK_LIBRARY PTHREADPOOL_LIBRARY) Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR) Could NOT find NCCL (missing: NCCL_INCLUDE_DIR NCCL_LIBRARY)

Could NOT find CUB (missing: CUB_INCLUDE_DIR) Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY)

Hiroki11x commented 6 years ago

the same error occured at ReedBush

-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found

similer issue https://github.com/caffe2/caffe2/issues/387 https://github.com/caffe2/caffe2/issues/866

areaChun commented 6 years ago

i have the same issue ,and can not resolve it.

CMakeFiles/cmTC_f9c29.dir/CheckSymbolExists.c.o:在函数‘main’中: CheckSymbolExists.c:(.text+0x16):对‘pthread_create’未定义的引用 collect2: error: ld returned 1 exit status CMakeFiles/cmTC_f9c29.dir/build.make:97: recipe for target 'cmTC_f9c29' failed make[2]: *** [cmTC_f9c29] Error 1 make[2]: Leaving directory '/home/zcr/caffe2/build/CMakeFiles/CMakeTmp' Makefile:126: recipe for target 'cmTC_f9c29/fast' failed

/usr/bin/ld: 找不到 -lpthreads collect2: error: ld returned 1 exit status CMakeFiles/cmTC_0c874.dir/build.make:97: recipe for target 'cmTC_0c874' failed make[2]: *** [cmTC_0c874] Error 1

Hiroki11x commented 6 years ago

In ReedBush(University of Tokyo Computing Resource)

I changed version of python and reinstall following

pip install numpy
pip install future
pip install protobuf

then, I edit install script (PYTHON_LIB,PYTHON_INCLUDE). I changed CMAKE_PREFIX_PATH with change of python as following due to protobuf.

git clone https://github.com/rioyokotalab/caffe2.git && cd caffe2
git submodule update --init

rm -rf build && mkdir build && cd build

#----- pyenv
export PYENV_ROOT="/lustre/gi75/i75012/env/src/pyenv"
if [ -d "${PYENV_ROOT}" ]; then
   export PATH=${PYENV_ROOT}/bin:$PATH
   eval "$(pyenv init -)"
fi

# load boost environment
module load boost/1.61 

LOCAL=/lustre/gi75/i75012/env/local
PYTHON_INCLUDE=/usr/include/python2.7:/lustre/gi75/i75012/env/src/pyenv/versions/2.7.9/lib/python2.7/site-packages/numpy/core/include
PYTHON_LIB=/lustre/gi75/i75012/env/src/pyenv/versions/2.7.9/lib/python2.7:/usr/lib
HDF5_HL_LIBRARIES=/lustre/gi75/i75012/env/local/hdf5-1.10.0-patch1/lib

declare -a PACKAGES=(\
'boost_1_63_0' \
'gflags-2.2.0' \
'glog-0.3.4' \
'hdf5-1.10.0-patch1' \
'lmdb-LMDB_0.9.18' \
'cudnn7/cuda' \
'snappy-1.1.4' \
'opencv-2.4.13' \
'nccl-1.3.4-1' \
'ATLAS' \
)

PREFIX_PATH="/lustre/gi75/i75012/env/local/cudnn7/cuda:/usr/local/cuda"

for s in "${PACKAGES[@]}"; do
PREFIX_PATH=$LOCAL/$s:$PREFIX_PATH
done

CMAKE_PREFIX_PATH=$PYTHON_INCLUDE:$PYTHON_LIB:$HDF5_HL_LIBRARIES:$PREFIX_PATH cmake \
-DCUDA_TOOLKIT_ROOT_DIR=/lustre/app/acc/cuda/8.0 \
-DCMAKE_INSTALL_PREFIX=/lustre/gi75/i75012/dl/caffe2/local \
-DUSE_NCCL=ON \
-DUSE_LEVELDB=OFF \
.. | tee configure.log

make all -j 248 && make test -j 248 && make install

then I succeeded to install like that https://github.com/rioyokotalab/caffe2/issues/21