search

vrenkens / nabu

Code for end-to-end ASR with neural networks, build with TensorFlow
MIT License
108 stars 43 forks source link

_cut_sequence issue #38

Closed rzcwade closed 6 years ago

rzcwade commented 6 years ago

Hi,

I was testing the cut sequence length in the recipe but ran into the following issue:

starting training Traceback (most recent call last): File "nabu/scripts/prepare_train.py", line 365, in main(FLAGS.expdir, FLAGS.recipe, FLAGS.mode, FLAGS.computing) File "nabu/scripts/prepare_train.py", line 90, in main expdir=expdir) File "/home/zichengr/Nabu_simple/nabu/scripts/train.py", line 85, in train tr.train(testing) File "/home/zichengr/Nabu_simple/nabu/neuralnetworks/trainers/trainer.py", line 603, in train outputs = self._create_graph() File "/home/zichengr/Nabu_simple/nabu/neuralnetworks/trainers/trainer.py", line 126, in _create_graph outputs['local_steps']) = self._data(chief_ps) File "/home/zichengr/Nabu_simple/nabu/neuralnetworks/trainers/trainer.py", line 376, in _data max_length) File "/home/zichengr/Nabu_simple/nabu/neuralnetworks/trainers/trainer.py", line 923, in _cut_sequence numcuts = tf.to_int32(tf.ceil(tf.shape(sequence)[1]/cut_length)) File "/home/zichengr/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1582, in ceil "Ceil", x=x, name=name) File "/home/zichengr/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 609, in _apply_op_helper param_name=input_name) File "/home/zichengr/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint ", ".join(dtypes.as_dtype(x).name for x in allowed_list))) TypeError: Value passed to parameter 'x' has DataType int32 not in list of allowed values: bfloat16, float16, float32, float64

Has it occurred to anyone?

Thanks

vrenkens commented 6 years ago

That functionality is quite untested, so I can imagine if it is not perfect. For what do you need it? I will be taking a new look at the code in the coming days and I will probably trim it down and try to make it simpler

rzcwade commented 6 years ago

Hi Vincent,

I just wanted to get a better understanding of your code. So I just tried different settings from the recipe. I actually fixed the issue raised from _cut_sequence function and now it worked fine for me.

Thanks

vrenkens commented 6 years ago

Just wanted you to know: there was a bug in the validation that caused the validation performance to average over all steps instead of replacing the performance wit the new validation performance. This gave the impression that the model converged very slowly. It has been fixed now and TIMIT training only takes a couple of hours now

rzcwade commented 6 years ago

Thanks for letting me know, Vincent!