PeiliangLi
commented
6 years ago I have the similar issue when I follow these steps https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_locally.md to train object detection. I use tensorflow object detection API and the model checkpoint is download from http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_kitti_2017_11_08.tar.gz. The pipeline config file and datasets are exactly same with the provide one. The train.py can work normally for several steps, and then segmentation fault. Sample logs:
INFO:tensorflow:global step 4457: loss = 0.0524 (0.523 sec/step)
INFO:tensorflow:global step 4458: loss = 0.0071 (0.789 sec/step)
INFO:tensorflow:global step 4459: loss = 0.4697 (0.677 sec/step)
INFO:tensorflow:global step 4460: loss = 0.0488 (0.396 sec/step)
INFO:tensorflow:global step 4461: loss = 0.0560 (0.637 sec/step)
INFO:tensorflow:global step 4462: loss = 0.0424 (0.588 sec/step)
INFO:tensorflow:global step 4463: loss = 0.0227 (0.525 sec/step)
INFO:tensorflow:global step 4464: loss = 0.0826 (0.693 sec/step)
Thread 157 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff7b4ff9700 (LWP 19993)]
0x00007fff4d5cdd80 in std::_Hashtable<std::string, std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*>, std::allocator<std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node(unsigned long, std::string const&, unsigned long) const ()
from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
(gdb) where
#0 0x00007fff4d5cdd80 in std::_Hashtable<std::string, std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*>, std::allocator<std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node(unsigned long, std::string const&, unsigned long) const ()
from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#1 0x00007fff4d5cde25 in std::_Hashtable<std::string, std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*>, std::allocator<std::pair<std::string const, std::unordered_map<std::pair<unsigned long long, std::string>, tensorflow::ResourceBase*, tensorflow::ResourceMgr::KeyHash, tensorflow::ResourceMgr::KeyEqual, std::allocator<std::pair<std::pair<unsigned long long, std::string> const, tensorflow::ResourceBase*> > >*> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find(std::string const&) const () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#2 0x00007fff4d5ce6e1 in tensorflow::ResourceMgr::DoLookup(std::string const&, std::type_index, std::string const&, tensorflow::ResourceBase**) const () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#3 0x00007fff4fbd36e6 in tensorflow::GetStack(tensorflow::OpKernelContext*, tensorflow::Stack**) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#4 0x00007fff4fbd439d in tensorflow::StackPushOp<Eigen::GpuDevice>::ComputeAsync(tensorflow::OpKernelContext*, std::function<void ()>) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#5 0x00007fff4da034ea in tensorflow::BaseGPUDevice::ComputeAsync(tensorflow::AsyncOpKernel*, tensorflow::OpKernelContext*, std::function<void ()>) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#6 0x00007fff4da3a17b in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#7 0x00007fff4da3a3ea in std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(tensorflow::gtl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8> const&, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#8 0x00007fff4d6d7782 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#9 0x00007fff4d6d6832 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#10 0x00007fff462dfc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007ffff7bc16ba in start_thread (arg=0x7ff7b4ff9700) at pthread_create.c:333
#12 0x00007ffff78f741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109System information
-
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
-
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.3 LTS (Xenial)
-
TensorFlow installed from (source or binary): Binary
-
TensorFlow version (use command below): ('v1.4.0-19-ga52c8d9', '1.4.1')
-
Python version: 2.7.12
-
CUDA/cuDNN version: Cuda8, cudnn6
-
GPU model and memory: Titan Xp (with driver 387.26)
tatatodd
mpkuse
skye
ghost
anhnt1
caot
mikeshiyang
mohantym
google-ml-butler[bot]
System information
You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
My custom learning code works perfectly on my older workstation with 2 GPU cards. But am having issue with our new workstation which has 8 GPU cards. I get a Segmentation fault.
Source code / logs
The entire source code is: https://github.com/mpkuse/cartwheel_train/tree/config-files
The main-script is
train_netvlad.py. Currently my learning data is private, if you really need it to test, I can provide the data as well (~100 GB).My code basically builds a network with tf.slim. I have a custom operations to build a layer. Have a custom loss function. Can be found in
CartWheelFlow.py/ class VGGDescriptor. It uses tf.while. Data is managed byclass TimeMachineRenderstack trace for the crash.