`Using all available GPUs.
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Enumerating dataset list from disk...
Found 5 videos for training.
Found 5 videos for validation.
Total params: 4,572,156.0
Trainable params: 4,571,772.0
Non-trainable params: 384.0
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.645
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.25GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
Epoch 0: Curriculum(train: True, sentence_length: -1, flip_probability: 0.5, jitter_probability: 0.05)
Epoch 1/5000
Epoch 0: Curriculum(train: True, sentence_length: -1, flip_probability: 0.5, jitter_probability: 0.05)
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds.
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds.
[[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]]
W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds.
[[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]]
Traceback (most recent call last):
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 77, in
train(run_name, 0, 5000, 3, 100, 50, 75, 32, 1)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 72, in train
pickle_safe=True)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 88, in wrapper
return func(*args, **kwargs)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/training.py", line 1876, in fit_generator
class_weight=class_weight)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/training.py", line 1620, in train_on_batch
outputs = self.train_function(ins)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2073, in call
feed_dict=feed_dict)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 0 of dimension 0 out of bounds.
[[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]]
[[Node: batc2/moments/sufficient_statistics/Gather/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_7459_batc2/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'ctc/scan/strided_slice', defined at:
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 77, in
train(run_name, 0, 5000, 3, 100, 50, 75, 32, 1)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 40, in train
absolute_max_string_len=absolute_max_string_len, output_size=lip_gen.get_output_size())
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/model2.py", line 21, in init
self.build()
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/model2.py", line 66, in build
self.loss_out = CTC('ctc', [self.y_pred, self.labels, self.input_length, self.label_length])
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/core/layers.py", line 7, in CTC
return Lambda(ctc_lambda_func, output_shape=(1,), name=name)(args)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/topology.py", line 554, in call
output = self.call(inputs, kwargs)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/layers/core.py", line 659, in call
return self.function(inputs, arguments)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/core/loss.py", line 11, in ctc_lambda_func
return K.ctc_batch_cost(labels, y_pred, input_length, label_length)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3258, in ctc_batch_cost
sparse_labels = tf.to_int32(ctc_label_dense_to_sparse(y_true, label_length))
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3222, in ctc_label_dense_to_sparse
initializer=init, parallel_iterations=1)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/functional_ops.py", line 524, in scan
n = array_ops.shape(elems_flat[0])[0]
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 495, in _SliceHelper
name=name)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 653, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3688, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): slice index 0 of dimension 0 out of bounds.
[[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]]
[[Node: batc2/moments/sufficient_statistics/Gather/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_7459_batc2/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
`
`Using all available GPUs. Using TensorFlow backend. I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Enumerating dataset list from disk... Found 5 videos for training. Found 5 videos for validation.
Layer (type) Output Shape Param #
the_input (InputLayer) (None, 75, 100, 50, 3) 0
zero1 (ZeroPadding3D) (None, 77, 104, 54, 3) 0
conv1 (Conv3D) (None, 75, 50, 25, 32) 7232
batc1 (BatchNormalization) (None, 75, 50, 25, 32) 128
actv1 (Activation) (None, 75, 50, 25, 32) 0
spatial_dropout3d_1 (Spatial (None, 75, 50, 25, 32) 0
max1 (MaxPooling3D) (None, 75, 25, 12, 32) 0
zero2 (ZeroPadding3D) (None, 77, 29, 16, 32) 0
conv2 (Conv3D) (None, 75, 25, 12, 64) 153664
batc2 (BatchNormalization) (None, 75, 25, 12, 64) 256
actv2 (Activation) (None, 75, 25, 12, 64) 0
spatial_dropout3d_2 (Spatial (None, 75, 25, 12, 64) 0
max2 (MaxPooling3D) (None, 75, 12, 6, 64) 0
zero3 (ZeroPadding3D) (None, 77, 14, 8, 64) 0
conv3 (Conv3D) (None, 75, 12, 6, 96) 165984
batc3 (BatchNormalization) (None, 75, 12, 6, 96) 384
actv3 (Activation) (None, 75, 12, 6, 96) 0
spatial_dropout3d_3 (Spatial (None, 75, 12, 6, 96) 0
max3 (MaxPooling3D) (None, 75, 6, 3, 96) 0
time_distributed_1 (TimeDist (None, 75, 1728) 0
bidirectional_1 (Bidirection (None, 75, 512) 3048960
bidirectional_2 (Bidirection (None, 75, 512) 1181184
dense1 (Dense) (None, 75, 28) 14364
softmax (Activation) (None, 75, 28) 0
Total params: 4,572,156.0 Trainable params: 4,571,772.0 Non-trainable params: 384.0
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate (GHz) 1.645 pciBusID 0000:01:00.0 Total memory: 7.92GiB Free memory: 7.25GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0) Epoch 0: Curriculum(train: True, sentence_length: -1, flip_probability: 0.5, jitter_probability: 0.05) Epoch 1/5000 Epoch 0: Curriculum(train: True, sentence_length: -1, flip_probability: 0.5, jitter_probability: 0.05) W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds. W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds. [[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]] W tensorflow/core/framework/op_kernel.cc:993] Invalid argument: slice index 0 of dimension 0 out of bounds. [[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]] Traceback (most recent call last): File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 77, in
train(run_name, 0, 5000, 3, 100, 50, 75, 32, 1)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 72, in train
pickle_safe=True)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 88, in wrapper
return func(*args, **kwargs)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/training.py", line 1876, in fit_generator
class_weight=class_weight)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/training.py", line 1620, in train_on_batch
outputs = self.train_function(ins)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2073, in call
feed_dict=feed_dict)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 0 of dimension 0 out of bounds.
[[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]]
[[Node: batc2/moments/sufficient_statistics/Gather/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_7459_batc2/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op u'ctc/scan/strided_slice', defined at: File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 77, in
train(run_name, 0, 5000, 3, 100, 50, 75, 32, 1)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/training/unseen_speakers/train.py", line 40, in train
absolute_max_string_len=absolute_max_string_len, output_size=lip_gen.get_output_size())
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/model2.py", line 21, in init
self.build()
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/model2.py", line 66, in build
self.loss_out = CTC('ctc', [self.y_pred, self.labels, self.input_length, self.label_length])
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/core/layers.py", line 7, in CTC
return Lambda(ctc_lambda_func, output_shape=(1,), name=name)(args)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/engine/topology.py", line 554, in call
output = self.call(inputs, kwargs)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/layers/core.py", line 659, in call
return self.function(inputs, arguments)
File "/home/deepakgupta1313/Desktop/Deepak/Programs/Github/LipNet/lipnet/core/loss.py", line 11, in ctc_lambda_func
return K.ctc_batch_cost(labels, y_pred, input_length, label_length)
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3258, in ctc_batch_cost
sparse_labels = tf.to_int32(ctc_label_dense_to_sparse(y_true, label_length))
File "/home/deepakgupta1313/anaconda3/envs/py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3222, in ctc_label_dense_to_sparse
initializer=init, parallel_iterations=1)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/functional_ops.py", line 524, in scan
n = array_ops.shape(elems_flat[0])[0]
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 495, in _SliceHelper
name=name)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 653, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3688, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
op_def=op_def)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2327, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/deepakgupta1313/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1226, in init
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): slice index 0 of dimension 0 out of bounds. [[Node: ctc/scan/strided_slice = StridedSlice[Index=DT_INT32, T=DT_INT32, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/gpu:0"](ctc/scan/Shape, ctc/scan/strided_slice/stack, ctc/scan/strided_slice/stack_1, ctc/scan/strided_slice/stack_2)]] [[Node: batc2/moments/sufficient_statistics/Gather/_155 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_7459_batc2/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]] `