Open tumusudheer opened 4 years ago
Also I trained the model with just 4 audio files of common_voice data , to give it a try. It has prepared encoder.subwords file and the vocab size is given as 301. I trained the model for 1000 epochs.
Now I want to train first 1000 audio files. When I used the command
python run_rnnt.py --mode train --data_dir data_trail/preprocessed_small_1000 --batch_size 8 --eval_size 100 --checkpoint model.bak.small_size_data/checkpoint_1000_826.4307.hdf5
, I'm getting the following error:
I believe this is due to new words got added; But for finetuing, the model is copying the old encounter.subwords from the checkpoint directory as here
When we want to to train with new data and want to finetune (restore weights from a trained checkpoint), how should we add new vocabulary of the new audio files transcriptions.
2020-06-12 15:05:46.302226: W tensorflow/core/framework/op_kernel.cc:1741] Invalid argument: ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
return func(device, token, args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
ret = self._func(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)
File "/tmp/tmpp084qsez.py", line 12, in <lambda>
retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
return _call_unconverted(f, args, kwargs, options, False)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
return f(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
return tf.constant(encoder.decode(ids.numpy()))
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
subword = self._id_to_subword(subword_id)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
"[0, %d)." % (subword_id + 1, self.vocab_size))
ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):
File "run_rnnt.py", line 590, in <module>
app.run(main)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "run_rnnt.py", line 549, in main
eval_metrics=[accuracy_fn, wer_fn])
File "run_rnnt.py", line 347, in run_training
checkpoint_model()
File "run_rnnt.py", line 312, in checkpoint_model
metrics=eval_metrics)
File "run_rnnt.py", line 436, in run_evaluate
loss, metrics_results = eval_step(inputs)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 650, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
self.captured_inputs)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 598, in call
ctx=ctx)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
return func(device, token, args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
ret = self._func(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)
File "/tmp/tmpp084qsez.py", line 12, in <lambda>
retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
return _call_unconverted(f, args, kwargs, options, False)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
return f(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
return tf.constant(encoder.decode(ids.numpy()))
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
subword = self._id_to_subword(subword_id)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
"[0, %d)." % (subword_id + 1, self.vocab_size))
ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
[[node EagerPyFunc (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py:40) ]]
(1) Invalid argument: ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
Traceback (most recent call last):
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
return func(device, token, args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
ret = self._func(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)
File "/tmp/tmpp084qsez.py", line 12, in <lambda>
retval_ = fscope.mark_return_value(ag__.converted_call(tf.py_function, ((lambda x: ag__.converted_call(wordpiece_decode, (x, encoder), None, fscope)),), dict(inp=[ids], Tout=[tf.string]), fscope)[0])
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 422, in converted_call
return _call_unconverted(f, args, kwargs, options, False)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py", line 348, in _call_unconverted
return f(*args)
File "/home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py", line 35, in wordpiece_decode
return tf.constant(encoder.decode(ids.numpy()))
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 113, in decode
subword = self._id_to_subword(subword_id)
File "/home/tumu/Self/Research/Work/tensorflow_work/tensorflow_2.2_env/lib/python3.6/site-packages/tensorflow_datasets/core/features/text/subword_text_encoder.py", line 179, in _id_to_subword
"[0, %d)." % (subword_id + 1, self.vocab_size))
ValueError: Received id 502 which is invalid. Ids must be within [0, 301).
[[node EagerPyFunc (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/encoding.py:40) ]]
[[StringSplit_1/StringSplit/StringSplitV2/_130]]
0 successful operations.
0 derived errors ignored. [Op:__inference_eval_step_5969]
Errors may have originated from an input operation.
Input Source operations connected to node EagerPyFunc:
strided_slice_4 (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/metrics.py:81)
Input Source operations connected to node EagerPyFunc:
strided_slice_4 (defined at /home/tumu/Self/Research/Work/tensorflow_work/models/try/rnnt-speech-recognition/utils/metrics.py:81)
Function call stack:
eval_step -> eval_step
Hi,
I would like to train my own dataset. Just wanted to know some guidelines on how to prepare data: I'll prepare some .wav files with 16KHz sample rate and single channel.
Also is a there a trained model I can start my training from instead of starting from scratch ?
Thanks.