search

bilylee / SiamFC-TensorFlow

A TensorFlow implementation of the SiamFC tracker
MIT License
363 stars 112 forks source link

when training: OutOfRange Error: end of sequence #40

Closed Francoforever closed 6 years ago

Francoforever commented 6 years ago

Excuse me, when I am executing the script Siamese-3s-color-scratch.py, it came up this problem: `INFO - SiamFC-3s-color-scratch - Running command 'main' INFO - SiamFC-3s-color-scratch - Started run with ID "22" INFO - root - Using largest free memory GPU 1 with free memory 11101.375MB INFO - root - preproces -- siamese_fc_color INFO - root - embedding init method -- kaiming_normal INFO - root - preproces -- None 2018-07-16 09:44:40.331186: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-07-16 09:44:41.609915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: name: Tesla K40c major: 3 minor: 5 memoryClockRate(GHz): 0.745 pciBusID: 0000:84:00.0 totalMemory: 11.17GiB freeMemory: 10.77GiB 2018-07-16 09:44:41.609956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:84:00.0, compute capability: 3.5) INFO - root - Train for 1250 steps 2018-07-16 09:44:44.885479: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:44.896464: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. INFO - root - 2018-07-16 09:44:46.990354: step 0, total loss = 3.11, batch loss = 2.54 (3.3 examples/sec; 2.413 sec/batch; 0h:50m:16s remains) 2018-07-16 09:44:47.238187: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.248992: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.251950: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.253937: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.255833: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.257882: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.259775: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.262073: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.264808: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.267238: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.269300: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.271260: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.273237: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.275208: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.277163: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.279094: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.281016: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.283020: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.285041: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.287280: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.289991: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.292109: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.294060: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.296047: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.298024: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.299885: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.301802: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.304281: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.306734: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.308728: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.310470: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.312440: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.314521: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.316580: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.318557: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.320619: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.322688: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.324736: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.326882: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.328834: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.330843: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.332869: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.334772: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.336784: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.338743: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.340699: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.342661: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.344552: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.346522: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.348528: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.350599: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.352681: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.354781: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.357392: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.359764: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.361748: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.363640: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.365568: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.367669: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.369718: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.372234: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.374938: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.377226: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.379163: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.381073: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.382975: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.384849: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.386896: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.388950: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.390904: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.392747: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.394711: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.396606: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.398464: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.400504: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.402437: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.404321: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.406220: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.408037: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.409904: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.411681: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.413505: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.415371: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.417238: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.419104: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.421027: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.422872: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.424850: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.426852: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.428696: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.430666: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.432671: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.434638: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.436560: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.438525: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.440986: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.443297: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.445647: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.448020: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.449949: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. 2018-07-16 09:44:47.452290: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished. ERROR - SiamFC-3s-color-scratch - Failed after 0:00:16! Traceback (most recent call last): File "experiments/SiamFC-3s-color-scratch.py", line 29, in '--enforce_clean': False, File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/experiment.py", line 209, in run run() File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/run.py", line 221, in call self.result = self.main_function(args) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/config/captured_function.py", line 46, in captured_function result = wrapped(args, **kwargs) File "experiments/../train_siamese_model.py", line 180, in main summary_str = sess.run(summary_op) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run run_metadata_ptr) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run feed_dict_tensor, options, run_metadata) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run options, run_metadata) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [[Node: validation/IteratorGetNext = IteratorGetNextoutput_shapes=[[?,127,127,3], [?,255,255,3]], output_types=[DT_UINT8, DT_UINT8], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] [[Node: validation/Loss/CD-err/auc/assert_less_equal/Assert/AssertGuard/Assert/_86 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_965_validation/Loss/CD-err/auc/assert_less_equal/Assert/AssertGuard/Assert", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op u'validation/IteratorGetNext', defined at: File "experiments/SiamFC-3s-color-scratch.py", line 29, in '--enforce_clean': False, File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/experiment.py", line 209, in run run() File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/run.py", line 221, in call self.result = self.main_function(args) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/sacred/config/captured_function.py", line 46, in captured_function result = wrapped(args, **kwargs) File "experiments/../train_siamese_model.py", line 106, in main model_va.build(reuse=True) File "experiments/../siamese_model.py", line 210, in build self.build_inputs() File "experiments/../siamese_model.py", line 62, in build_inputs exemplars, instances = self.dataloader.get_one_batch() File "experiments/../datasets/dataloader.py", line 103, in get_one_batch return self.iterator.get_next() File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 259, in get_next name=name)) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 706, in iterator_get_next output_shapes=output_shapes, name=name) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op op_def=op_def) File "/home/jiangshan/.conda/envs/py2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): End of sequence [[Node: validation/IteratorGetNext = IteratorGetNextoutput_shapes=[[?,127,127,3], [?,255,255,3]], output_types=[DT_UINT8, DT_UINT8], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] [[Node: validation/Loss/CD-err/auc/assert_less_equal/Assert/AssertGuard/Assert/_86 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_965_validation/Loss/CD-err/auc/assert_less_equal/Assert/AssertGuard/Assert", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]`

bilylee commented 6 years ago

Hi,

It seems like you don't have enough validation data. Please check by

import os.path as osp
import pickle
with open('data/validation_imdb.pickle') as f:
    imdb = pickle.load(f)

n_frames = 0
for v in imdb['videos']:
    n_frames += len(v)

total_file_size = 0 # Byte
for v in imdb['videos']:
    for p in v:
        total_file_size += osp.getsize(p)
total_file_size /= (1024 * 1024 * 1024) # GB

print('Num of videos: {}'.format(imdb['n_videos']))
print('Num of frames: {}'.format(n_frames))
print('Total file size: {} GB'.format(total_file_size))
bilylee commented 6 years ago

closing due to lack of response. Feel free to reopen. : )