search

cdyangbo / end2endASR

implement end-to-end asr algorithm with tensorflow
40 stars 22 forks source link

You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape #6

Open GabrielLin opened 6 years ago

GabrielLin commented 6 years ago

When I train to epoch 3, the following errors have been shown. Could you help? thanks.

===========master initModelParams================= fail to restore model params from /data1/asr/savemodels/ds2_0311 Epoch 0: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [05:26<00:00, 2.39s/it, loss=90.9, refresh=0, step=61] Epoch 1: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:32<00:00, 6.07s/it, loss=249, refresh=0, step=122] Epoch 2: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:09<00:00, 5.94s/it, loss=256, refresh=0, step=183] Epoch 3: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:06<00:00, 6.32s/it, loss=269, refresh=0, step=244] 06-11 15:49:23 Epoch:3,begin validation, batch_num:66 2018-06-11 15:49:23.917666: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.917668: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.918618: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.918721: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.918824: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.918889: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.918947: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919004: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919060: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919127: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919185: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919246: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919303: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919359: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919411: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919467: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919520: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919574: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919628: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919680: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919735: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919788: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919843: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919899: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.919951: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920006: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920058: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920117: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920177: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920230: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920506: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920581: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920640: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920698: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920755: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920810: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920866: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920921: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.920977: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921029: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921085: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921147: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921202: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921258: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921574: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921655: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2018-06-11 15:49:23.921717: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] Traceback (most recent call last): File "run_train.py", line 66, in tr.train() File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 163, in train self.train_single() File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 180, in train_single self.run_train_epoch(sess,epoch+self.args.initial_epoch,self.args.batch_size) File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 269, in run_train_epoch decode_cost,decode_cer, decode_wer = self.run_validate_decode_epoch(sess, epoch, batch_size) File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 328, in run_validate_decode_epoch feed_dict=feed_dict) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run feed_dict_tensor, options, run_metadata) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run options, run_metadata) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] [[Node: bidrnn4/bw/bw/All/_1767 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_259_bidrnn4/bw/bw/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'target_seq_length', defined at: File "run_train.py", line 59, in tr.build_model() File "/data1/myname/nlp/end2endASR/utils/misc.py", line 78, in wrapper result = func(*args, **kwargs) File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 83, in build_model self.classifier = model_factory(args=self.args, server=self.server, device=self.device) File "/data1/myname/nlp/end2endASR/model/model_factory.py", line 35, in model_factory return ds2_classifier.DeepSpeech2(args, name or args.model, server=server, device=device) File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 134, in init self.build_graph() File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 208, in build_graph self.add_input_layer() File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 152, in add_input_layer name='target_seq_length') File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1548, in placeholder return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2094, in _placeholder name=name) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op original_op=self._default_original_op, op_def=op_def) File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32] [[Node: target_seq_length = Placeholder[dtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]()]] [[Node: bidrnn4/bw/bw/All/_1767 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_259_bidrnn4/bw/bw/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

poor1017 commented 5 years ago

Hi, I encounter the same issue, have you found the solution? Thank you.

GabrielLin commented 5 years ago

@poor1017 Not yet, I hope you can share your solution if you find it. Thanks.

poor1017 commented 5 years ago

I think you used ds2 model, so you need add two lines:

@@ -134,6 +134,9 @@ class Trainer(object): feed_dict[self.classifier.targetVals] = target_vals feed_dict[self.classifier.targetShape] = target_shape feed_dict[self.classifier.input_seq_length] = seqlens

cdyangbo commented 5 years ago

this bug may be fix soon! in ds2 model don't need this parameters, just omit it .