tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. <eos>]

[nipengmath]

Hi Vincent! I got a problem when i run the recipe LAS/TIMIT, can you give me some help to fix this error? Thanks !

File "/data/nipeng/nabu/nabu/neuralnetworks/trainers/trainer.py", line 819, in train outputs['training_summaries']]) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 695, in exit self._close_internal(exception_type) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 732, in _close_internal self._sess.close() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in close self._sess.close() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1124, in close ignore_live_threads=True) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run enqueue_callable() File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run self._call_tf_sessionrun(None, {}, [], target_list, None) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. ]

[vrenkens]

Hi!

The recipe is for phonetic outputs, but you are using textual ones.

You’re target file should contain the phonetic transcriptions.

Cheers, Vincent

On Wed, 23 Jan 2019 at 08:39, Ni Peng notifications@github.com wrote:

Hi Vincent! I got a problem when i run the recipe LAS/TIMIT, can you give me some help to fix this error? Thanks !

File "/data/nipeng/nabu/nabu/neuralnetworks/trainers/trainer.py", line 819, in train outputs['training_summaries']]) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 695, in exit self._close_internal(exception_type) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 732, in _close_internal self._sess.close() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in close self._sess.close() File "/usr/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1124, in close ignore_live_threads=True) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/usr/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run enqueue_callable() File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1205, in _single_operation_run self._call_tf_sessionrun(None, {}, [], target_list, None) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [Jeff thought you argued in favor of a centrifuge purchase. ]

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/vrenkens/nabu/issues/55, or mute the thread https://github.com/notifications/unsubscribe-auth/APTcBq3yR4eKo3yMIQla36KRb0sMCClHks5vGBG8gaJpZM4aONCo .

[jtcampbellsoup]

Hi Vincent,

I'm running into a similar problem to this, I believe. Can you clarify what you mean by setting the target file to contain the phonetic transcriptions?

I'm using the DBLSTM/TIMIT recipe, and the only changes I've made to it are to the database.cfg file, where I set the data files to the proper locations. So the wave files were processed by the feature_processor.cfg and the text was processed by the text_processor.cfg successfully. (I am using a dataset of isolated-one word utterances and their transcriptions from people with dysarthria).

Now, when I run: ./run train --recipe=config/recipes/DBLSTM/TIMIT --expdir=Models/DBLSTM_01 --mode=non_distributed --computing=standard

I get the following traceback error:

Traceback (most recent call last): File "nabu/scripts/prepare_train.py", line 366, in main(FLAGS.expdir, FLAGS.recipe, FLAGS.mode, FLAGS.computing) File "nabu/scripts/prepare_train.py", line 91, in main expdir=expdir) File "/home/ubuntu/cs230project/SR-dysarthria/nab/nabu/scripts/train.py", line 90, in train tr.train(testing) File "/home/ubuntu/cs230project/SR-dysarthria/nab/nabu/neuralnetworks/trainers/trainer.py", line 787, in train outputs['increment_step'].run(session=sess) File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 788, in exit self._close_internal(exception_type) File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 826, in _close_internal self._sess.close() File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1082, in close self._sess.close() File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1244, in close ignore_live_threads=True) File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join six.reraise(*self._exc_info_to_raise) File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 257, in _run enqueue_callable() File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1257, in _single_operation_run self._call_tf_sessionrun(None, {}, [], target_list, None) File "/home/ubuntu/anaconda3/envs/tensorflow_p27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [behavior] [[{{node validate/validation/evaluate/input_pipeline/read_data/reader_1/StringReader/assert_equal/Assert/Assert}}]]

Any help by either of you would be greatly appreciated!

[vrenkens]

Hey, TIMIT has both textual (written) transcriptions and phonetic transcriptions (sequences of phones). The textual transcriptions are just text, meaning we can read it. The phonetic transcriptions contain phones (symbols like sh aa ae...). The TIMIT recipe for Nabu counts on you using the phonetic transcriptions. It's been a bit too long for me to remember what the path to the phonetic transcriptions typically is in TIMIT... You can probably find some more information by googling it or just exploring the files

[jtcampbellsoup]

Hmmm, I apologize but I am still a bit confused by this. In the DBLSTM/TIMIT recipe's database.cfg file, it specifies to use the text_processor on the text data. The text_processor.cfg file looks like this:

So it's taking in text data, and processing it as phoenetic transcriptions, right? Which are then used for training. I've done this pre-processing and the files were saved to the location specified in the database.cfg file. So why is the word "behavior" still showing up in the error above? Should that not be the case?

Really appreciate your help on this. The flexibility of this repository is fantastic!

[vrenkens]

Hi, the term text_processor is probably a bit confusing. In this case it should actually be processing phones. So it should not be taking in text data, it should be taking in phone data. Could maybe show the data files you have available for limit, what you have defined in your database.cfg file and the content of these files?

[jtcampbellsoup]

Here is the database.cfg file in that recipe:

Here is the train.scp file that it points to, where each line matches a label to a path to a wave file on my local computer:

Here is the train.txt file that it points to, where each line matches a label to the transcript of the corresponding wave file:

It seems perhaps that the the train.txt file is incorrect, and should be listing phones instead? And if so, does the "decoder" part of the LSTM network decode the information back to phones (and therefore output phones) or does it output full words?

[vrenkens]

It should indeed be listing phones. The decoder outputs phones, not full words.

[jtcampbellsoup]

Any recommendations of how I can decompose my words into phones?

[vrenkens]

Usually this is done when you do the Kaldi data prep for TIMIT