search

BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.06k stars 18.7k forks source link

make runtest failed #5077

Closed kkarnatak closed 7 years ago

kkarnatak commented 7 years ago

I have the following error while running the 'make runtest'. I have tried with cpu flag enabled and disabled both, same error.

----------] 5 tests from ImageDataLayerTest/1, where TypeParam = caffe::CPUDevice<double> [ RUN ] ImageDataLayerTest/1.TestReshape [ OK ] ImageDataLayerTest/1.TestReshape (30 ms) [ RUN ] ImageDataLayerTest/1.TestShuffle [ OK ] ImageDataLayerTest/1.TestShuffle (113 ms) [ RUN ] ImageDataLayerTest/1.TestRead [ OK ] ImageDataLayerTest/1.TestRead (114 ms) [ RUN ] ImageDataLayerTest/1.TestResize *** Aborted at 1481237980 (unix time) try "date -d @1481237980" if you are using GNU date *** PC: @ 0x7ffb7938b0ba (unknown) *** SIGSEGV (@0xfffffffffffffff7) received by PID 5373 (TID 0x7ffb62eaa700) from PID 18446744073709551607; stack trace: *** @ 0x7ffb8254d670 (unknown) @ 0x7ffb7938b0ba (unknown) @ 0x7ffb7938b18b (unknown) @ 0x7ffb7938cce8 (unknown) @ 0x7ffb7938b692 (unknown) @ 0x7ffb79386020 (unknown) @ 0x7ffb79384165 tbb::internal::allocate_root_with_context_proxy::allocate() @ 0x7ffb80db6e22 cv::parallel_for_() @ 0x7ffb8117ab2a (unknown) @ 0x7ffb81177bb7 cv::resize() @ 0x7ffb83061997 caffe::ReadImageToCVMat() @ 0x7ffb82f708bb caffe::ImageDataLayer<>::load_batch() @ 0x7ffb82efc906 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry() @ 0x7ffb82984596 (unknown) @ 0x7ffb8254370a start_thread @ 0x7ffb801b40af clone @ 0x0 (unknown) Segmentation fault (core dumped) src/caffe/test/CMakeFiles/runtest.dir/build.make:57: recipe for target 'src/caffe/test/CMakeFiles/runtest' failed

cdluminate commented 7 years ago

What's your operating system? Ubuntu devel seems to suffer from a similar problem.

https://launchpadlibrarian.net/298212926/buildlog_ubuntu-zesty-amd64.caffe_1.0.0~rc3+20161127-g24d2f67-4_BUILDING.txt.gz

kkarnatak commented 7 years ago

I am using kubuntu 16.10

On 16 Dec 2016 02:18, "L. Zhou" notifications@github.com wrote:

What's your operating system? Ubuntu devel seems to suffer from a similar problem.

https://launchpadlibrarian.net/298212926/buildlog_ubuntu- zesty-amd64.caffe_1.0.0~rc3+20161127-g24d2f67-4_BUILDING.txt.gz https://launchpadlibrarian.net/298212926/buildlog_ubuntu-zesty-amd64.caffe_1.0.0%7Erc3+20161127-g24d2f67-4_BUILDING.txt.gz

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/BVLC/caffe/issues/5077#issuecomment-267492285, or mute the thread https://github.com/notifications/unsubscribe-auth/AH_dXyuJKIfFjQfHQKQklEM7PubSEYf1ks5rIebQgaJpZM4LIYwD .

stjordanis commented 7 years ago

I also have ubuntu 16.10, no cpu.

I have the same issue too, although:

After I set DEBUG=1, and I ran:

protoc src/caffe/proto/caffe.proto --cpp_out=. mkdir include/caffe/proto mv src/caffe/proto/caffe.pb.h include/caffe/proto

because I had issues with caffe.pb.h,

runtest fails after a while when running: make runtest

I get (excluding some needless lines): .build_debug/tools/caffe .build_debug/test/test_all.testbin 0 --gtest_shuffle --gtest_filter="-GPU" Note: Google Test filter = -GPU Note: Randomizing tests' orders with a seed of 77085 .

Many test pass but then: [----------] 5 tests from ImageDataLayerTest/1, where TypeParam = caffe::CPUDevice [ RUN ] ImageDataLayerTest/1.TestRead [ OK ] ImageDataLayerTest/1.TestRead (84 ms) [ RUN ] ImageDataLayerTest/1.TestResize Aborted at 1481935191 (unix time) try "date -d @1481935191" if you are using GNU date PC: @ 0x7f14a0c870ba (unknown) SIGSEGV (@0xfffffffffffffff7) received by PID 16487 (TID 0x7f14871f2700) from PID 18446744073709551607; stack trace: @ 0x7f14a2152630 (unknown) @ 0x7f14a0c870ba (unknown) @ 0x7f14a0c8718b (unknown) @ 0x7f14a0c88ce8 (unknown) @ 0x7f14a0c87692 (unknown) @ 0x7f14a0c82020 (unknown) @ 0x7f14a0c80165 tbb::internal::allocate_root_with_context_proxy::allocate() @ 0x7f14a3e50e22 cv::parallelfor() @ 0x7f14a3737b2a (unknown) @ 0x7f14a3734bb7 cv::resize() @ 0x7f14a2c093dd caffe::ReadImageToCVMat() @ 0x7f14a2d0694c caffe::ImageDataLayer<>::load_batch() @ 0x7f14a2c9037c caffe::BasePrefetchingDataLayer<>::InternalThreadEntry() @ 0x7f14a2c2e19b caffe::InternalThread::entry() @ 0x7f14a2c307d1 boost::_mfi::mf5<>::operator()() @ 0x7f14a2c306a9 boost::_bi::list6<>::operator()<>() @ 0x7f14a2c3059e boost::_bi::bind_t<>::operator()() @ 0x7f14a2c30550 boost::detail::thread_data<>::run() @ 0x7f14a33f7596 (unknown) @ 0x7f14a21486ca start_thread @ 0x7f14a1e820af clone @ 0x0 (unknown) Segmentation fault (core dumped)


My config file:

# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
 CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
 #USE_OPENCV := 1
 #USE_LEVELDB := 1
 #USE_LMDB := 0

# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
        -gencode arch=compute_20,code=sm_21 \
        -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=sm_50 \
        -gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 \
#       /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
 PYTHON_INCLUDE :=  /home/steliox/anaconda2/include/python2.7 \
  /home/steliox/anaconda2/include \
         /home/steliox/anaconda2/lib/python2.7/site-packages/numpy/core/include \

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
#PYTHON_LIB := /usr/lib
 PYTHON_LIB := /home/steliox/anaconda2/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS :=  /usr/local/include /usr/include/hdf5/serial/
LIBRARY_DIRS :=  /usr/local/lib /usr/lib  /usr/lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/hdf5/serial/

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
 DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @
mmderakhshani commented 7 years ago

Hi, I would like to know, Is your problem solved? when I run make runset,I have got a same issue on ubuntu 16.04! could you please help me?

stjordanis commented 7 years ago

@mohammad-py It still fails in the same way exactly as mentioned above... I just cloned the latest caffe git repo + I have been updating my ubuntu system normally, running v17 now: I made the same procedure, but.... when I tried make runtest the same things happen.

svanschalkwyk commented 7 years ago

I get Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version Ubuntu 16.10 (Mint), gcc5, CUDA 8, Driver 378

rongfeng-china commented 7 years ago

I installed boost1.58 and added linked library and include path in bashrc. Then I had an error running ‘make runtest', and finally I found that I made a tiny mistake in bashrc.

The boost lib should be added to LD_LIBRARY, rather than LIBRARY...

kkarnatak commented 7 years ago

Hi Guys,

I started using https://github.com/BVLC/caffe/tree/opencl

and now I am able to successfully execute make runtest. I am using opencl mode. I havent tried CPU only mode but if you have AMD this will work provided you have successfully installed AMD graphic card. I had to downgrade to 14.04 to install AMD graphic card.

shelhamer commented 7 years ago

From https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md:

Please do not post usage, installation, or modeling questions, or other requests for help to Issues. Use the caffe-users list instead. This helps developers maintain a clear, uncluttered, and efficient view of the state of Caffe.

fan0916 commented 7 years ago

hello all i have issued this today, and finally i've solved this by changing libtbb2 version i'm using Ubuntu 16.04 and it uses libtbb2 (4.4~20160526-0ubuntu1) [universe] as default from belowlink i found that there is a bug in pre-build package https://software.intel.com/en-us/forums/intel-threading-building-blocks/topic/636519 so i changed to a new libtbb2 package as follow: libtbb2 (4.4~20151115-0ubuntu3) [universe] http://packages.ubuntu.com/xenial/libtbb2 this solved my problem regards!