INFO - root - 2019-03-08 23:15:09.660454: step 880, total loss = 1.54, batch loss = 1.02 (29.3 examples/sec; 0.273 sec/batch; 25h:10m:31s remains)
2019-03-08 23:15:11.022697: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: exceptions.ValueError: a must be non-empty
2019-03-08 23:15:11.128763: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: Iteration finished.
2019-03-08 23:15:11.286403: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: exceptions.ValueError: a must be non-empty
[[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_STRING], token="pyfunc_1"](arg0)]]
2019-03-08 23:15:11.288081: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: exceptions.ValueError: a must be non-empty
[[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_STRING], token="pyfunc_1"](arg0)]]
[[Node: train/IteratorGetNext = IteratorGetNext[output_shapes=[[?,127,127,3], [?,239,239,3]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](train/OneShotIterator)]]
2019-03-08 23:15:11.288117: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: exceptions.ValueError: a must be non-empty
[[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_STRING], token="pyfunc_1"](arg0)]]
[[Node: train/IteratorGetNext = IteratorGetNext[output_shapes=[[?,127,127,3], [?,239,239,3]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](train/OneShotIterator)]]
ERROR - SiamFC-3s-color-scratch - Failed after 0:06:42!
Traceback (most recent call last):
File "experiments/SiamFC-3s-color-scratch.py", line 29, in <module>
'--enforce_clean': False,
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/sacred/experiment.py", line 209, in run
run()
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/sacred/run.py", line 221, in __call__
self.result = self.main_function(*args)
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/sacred/config/captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "experiments/../train_siamese_model.py", line 166, in main
_, loss, batch_loss = sess.run([train_op, model.total_loss, model.batch_loss])
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/xieyangyang/anaconda3/envs/tensorflow1/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: exceptions.ValueError: a must be non-empty
[[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_STRING], token="pyfunc_1"](arg0)]]
[[Node: train/IteratorGetNext = IteratorGetNext[output_shapes=[[?,127,127,3], [?,239,239,3]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](train/OneShotIterator)]]