search

mozilla / DeepSpeech

DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.
Mozilla Public License 2.0
25.03k stars 3.94k forks source link

ValueError: Input DebugInfo is invalid. #2414

Closed MetaDhanesh closed 4 years ago

MetaDhanesh commented 4 years ago

I am using DeepSpeech to create a tflite model for Google's Speech Commands Dataset

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" 


Command:

DeepSpeech.py --n_hidden 256 --learning_rate 0.00001 --epochs 1 --export_tflite --export_dir /data/modelgen --train_files ../training65kgoogle/train_open.csv --dev_files ../training65kgoogle/dev_open.csv --test_files ../training65kgoogle/test_open.csv


When running the above command, I get ValueError. Error trace:

/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /libs/base/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) WARNING:tensorflow:From ./DeepSpeech.py:832: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /data/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1008 10:30:55.607139 140310328948544 deprecation_wrapper.py:119] From /data/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From ./DeepSpeech.py:814: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

W1008 10:30:55.613055 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:814: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From ./DeepSpeech.py:815: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W1008 10:30:55.613335 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:815: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2.

W1008 10:30:55.958150 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2.

WARNING:tensorflow:From /data/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

W1008 10:30:55.985267 140310328948544 deprecation_wrapper.py:119] From /data/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From ./DeepSpeech.py:375: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

W1008 10:30:56.040068 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:375: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From ./DeepSpeech.py:375: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_types(dataset). W1008 10:30:56.040287 140310328948544 deprecation.py:323] From ./DeepSpeech.py:375: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_types(dataset). WARNING:tensorflow:From ./DeepSpeech.py:376: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_shapes(dataset). W1008 10:30:56.040420 140310328948544 deprecation.py:323] From ./DeepSpeech.py:376: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_shapes(dataset). WARNING:tensorflow:From ./DeepSpeech.py:377: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_classes(dataset). W1008 10:30:56.040525 140310328948544 deprecation.py:323] From ./DeepSpeech.py:377: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_classes(dataset). WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_types(iterator). W1008 10:30:56.042337 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_types(iterator). WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_shapes(iterator). W1008 10:30:56.042538 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_shapes(iterator). WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_classes(iterator). W1008 10:30:56.042642 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.compat.v1.data.get_output_classes(iterator). WARNING:tensorflow:From ./DeepSpeech.py:388: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W1008 10:30:56.203413 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:388: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From ./DeepSpeech.py:211: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W1008 10:30:56.206955 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:211: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W1008 10:30:56.405591 140310328948544 deprecation.py:506] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From ./DeepSpeech.py:192: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

W1008 10:30:57.167075 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:192: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W1008 10:30:57.219491 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From ./DeepSpeech.py:273: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1008 10:30:57.765024 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:273: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From ./DeepSpeech.py:332: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W1008 10:30:57.839806 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:332: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From ./DeepSpeech.py:410: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W1008 10:30:58.045093 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:410: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From ./DeepSpeech.py:414: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W1008 10:30:58.191591 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:414: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From ./DeepSpeech.py:416: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W1008 10:30:58.192507 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:416: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From ./DeepSpeech.py:421: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1008 10:30:58.200566 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:421: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. W1008 10:30:58.309313 140310328948544 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/train-229122 I1008 10:30:58.323175 140310328948544 saver.py:1280] Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/train-229122 WARNING:tensorflow:From ./DeepSpeech.py:357: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W1008 10:30:59.173612 140310328948544 deprecation_wrapper.py:119] From ./DeepSpeech.py:357: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

I Restored variables from most recent checkpoint at /home/jupyter/.local/share/deepspeech/checkpoints/train-229122, step 229122 I STARTING Optimization Epoch 0 | Training | Elapsed Time: 0:05:12 | Steps: 22740 | Loss: 0.485720
Epoch 0 | Validation | Elapsed Time: 0:00:29 | Steps: 6088 | Loss: 0.975475 | Dataset: ../training65kgoogle/dev_open.csv I Saved new best validating model with loss 0.975475 to: /home/jupyter/.local/share/deepspeech/checkpoints/best_dev-251862 I FINISHED optimization in 0:05:43.925413 INFO:tensorflow:Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/best_dev-251862 I1008 10:36:43.868929 140310328948544 saver.py:1280] Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/best_dev-251862 I Restored variables from best validation checkpoint at /home/jupyter/.local/share/deepspeech/checkpoints/best_dev-251862, step 251862 Testing model on ../training65kgoogle/test_open.csv Test epoch | Steps: 7064 | Elapsed Time: 0:06:58
Test on ../training65kgoogle/test_open.csv - WER: 0.103766, CER: 0.069868, loss: 0.945907

WER: 1.000000, CER: 0.333333, loss: 0.009082

In the above case, training is successful, however during model export to tflite, throws error.

lissyx commented 4 years ago

deepspeech-gpu version: 0.5.1

You don't need that

lissyx commented 4 years ago

/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/lite/python/lite.py:769: UserWarning: Property post_training_quantize is deprecated, please use optimizations=[Optimize.DEFAULT] instead. " instead." % name) Traceback (most recent call last): File "./DeepSpeech.py", line 832, in tf.app.run(main) File "/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/jupyter/.local/lib/python3.6/site-packages/absl/app.py", line 299, in run _run_main(main, args) File "/home/jupyter/.local/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "./DeepSpeech.py", line 824, in main export() File "./DeepSpeech.py", line 753, in export tflite_model = converter.convert() File "/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/lite/python/lite.py", line 898, in convert **converter_kwargs) File "/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/lite/python/convert.py", line 404, in toco_convert_impl input_data.SerializeToString()) File "/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/lite/python/convert.py", line 172, in toco_convert_protos "TOCO failed. See console for info.\n%s\n%s\n" % (stdout, stderr)) tensorflow.lite.python.convert.ConverterError: TOCO failed. See console for info. /libs/base/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Traceback (most recent call last): File "/home/jupyter/libraries/nb_env/bin/toco_from_protos", line 11, in sys.exit(main()) File "/home/jupyter/libraries/nb_env/lib64/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 89, in main app.run(main=execute, argv=[sys.argv[0]] + unparsed) File "/home/jupyter/libraries/nb_env/lib64/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/libs/project/absl/app.py", line 299, in run _run_main(main, args) File "/libs/project/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "/home/jupyter/libraries/nb_env/lib64/python3.6/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 52, in execute enable_mlir_converter) ValueError: Input DebugInfo is invalid.

Obviously never hit that. I see references to Jupyter. Can you reproduce that with a pure Python virtualenv and no other overlay / wrapper?

lissyx commented 4 years ago

@MetaDhanesh It's working on my system when using LDC93S1 sample:

(tf-venv) alex@portable-alex:~/codaz/Mozilla/DeepSpeech/deepspeech-kdavis$ python DeepSpeech.py --n_hidden 256 --learning_rate 0.00001 --epochs 1 --export_tflite --export_dir modelgen --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv  --test_files data/ldc93s1/ldc93s1.csv                                                                                       [1/1887]
W1009 12:04:24.991541 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.                                                                                                             
Instructions for updating:                                                
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W1009 12:04:25.041713 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W1009 12:04:25.041825 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W1009 12:04:25.041942 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W1009 12:04:25.484676 140319504627520 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1009 12:04:25.486298 140319504627520 deprecation.py:506] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1009 12:04:25.944385 140319504627520 deprecation.py:323] From DeepSpeech.py:232: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1009 12:04:26.372065 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I1009 12:04:26.373267 140319504627520 saver.py:1280] Restoring parameters from /home/alex/.local/share/deepspeech/checkpoints/train-1
I Restored variables from most recent checkpoint at /home/alex/.local/share/deepspeech/checkpoints/train-1, step 1
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 354.431152
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 352.015808 | Dataset: data/ldc93s1/ldc93s1.csv
I Saved new best validating model with loss 352.015808 to: /home/alex/.local/share/deepspeech/checkpoints/best_dev-2
I FINISHED optimization in 0:00:00.635227
I1009 12:04:27.238729 140319504627520 saver.py:1280] Restoring parameters from /home/alex/.local/share/deepspeech/checkpoints/best_dev-2
I Restored variables from best validation checkpoint at /home/alex/.local/share/deepspeech/checkpoints/best_dev-2, step 2
Testing model on data/ldc93s1/ldc93s1.csv
Test epoch | Steps: 1 | Elapsed Time: 0:00:00
Test on data/ldc93s1/ldc93s1.csv - WER: 1.000000, CER: 0.846154, loss: 352.015808
--------------------------------------------------------------------------------
WER: 1.000000, CER: 0.846154, loss: 352.015808
 - wav: file:///home/alex/codaz/Mozilla/DeepSpeech/deepspeech-kdavis/data/ldc93s1/LDC93S1.wav
 - src: "she had your dark suit in greasy wash water all year"
 - res: "cincinnati c'incominciammo chachachachacha"
--------------------------------------------------------------------------------
I Exporting the model...
W1009 12:04:27.707445 140319504627520 deprecation_wrapper.py:119] From DeepSpeech.py:678: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

W1009 12:04:27.740257 140319504627520 deprecation.py:323] From DeepSpeech.py:128: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W1009 12:04:27.767323 140319504627520 deprecation.py:323] From DeepSpeech.py:138: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell, unroll=True)`, which is equivalent to this API
W1009 12:04:27.773589 140319504627520 deprecation.py:506] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py:961: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
I1009 12:04:28.316096 140319504627520 saver.py:1280] Restoring parameters from /home/alex/.local/share/deepspeech/checkpoints/train-2
W1009 12:04:28.344184 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W1009 12:04:28.344364 140319504627520 deprecation.py:323] From /home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
I1009 12:04:28.362188 140319504627520 graph_util_impl.py:311] Froze 12 variables.
I1009 12:04:28.366831 140319504627520 graph_util_impl.py:364] Converted 12 variables to const ops.
/home/alex/codaz/Mozilla/DeepSpeech/tf-venv/lib/python3.7/site-packages/tensorflow/lite/python/lite.py:769: UserWarning: Property post_training_quantize is deprecated, please use optimizations=[Optimize.DEFAULT] instead.
  " instead." % name)
I Exported model for TF Lite engine as output_graph.tflite
I Models exported at modelgen

I'm going to test your dataset, can you cross-check on your side with LDC93S1 sample? See bin/import_ldc93s1.py and data/ldc93s1/ldc93s1.csv.

lissyx commented 4 years ago

@MetaDhanesh It looks like the CSV files you are referencing in your issue do not exists in the dataset released by Google at https://ai.googleblog.com/2017/08/launching-speech-commands-dataset.html

Can you please share complete STR so I can check whether this is from the dataset itself (I highly doubt it is ...).

MetaDhanesh commented 4 years ago

training_csv_files.zip

I am storing the audio files and csv files (train, dev, test) in /data/training65kgoogle

lissyx commented 4 years ago

Have you reproduced the bug with ldc93s1 as I asked?

lissyx commented 4 years ago

@MetaDhanesh Please, take a few minutes to reproduce the issue with LDC93S1, it is very important to ensure your system works as intended.

The error you are facing is from within LLVM tooling of the TOCO converter, so it's very very highly unlikely that our code is at fault here.

lissyx commented 4 years ago

training_csv_files.zip

I am storing the audio files and csv files (train, dev, test) in /data/training65kgoogle

I'm sorry, but those CSV files seems not to be for this dataset, it does not contains the same directories, and files referenced inside do not exists in the linked data.

@MetaDhanesh I really need you to share me reproductible data. And to cross-check on your side with the LDC93S1 sample, to verify that you can perform the export.

MetaDhanesh commented 4 years ago

@lissyx As suggested by you, I ran the following command:

./DeepSpeech.py --n_hidden 256 --learning_rate 0.00001 --epochs 1 --export_tflite --export_dir /data/modelgen_ldc93s1 --train_files /data/ldc93s1/ldc93s1.csv --dev_files /data/ldc93s1/ldc93s1.csv  --test_files /data/ldc93s1/ldc93s1.csv

However, I got another issue (ZeroDivisionError):

/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/libs/base/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/jupyter/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From ./DeepSpeech.py:832: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /data/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1010 05:29:49.026217 140177034217280 deprecation_wrapper.py:119] From /data/DeepSpeech/util/config.py:60: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From ./DeepSpeech.py:814: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

W1010 05:29:49.031841 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:814: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From ./DeepSpeech.py:815: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W1010 05:29:49.032057 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:815: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W1010 05:29:49.051225 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

WARNING:tensorflow:From /data/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

W1010 05:29:49.079966 140177034217280 deprecation_wrapper.py:119] From /data/DeepSpeech/util/feeding.py:45: The name tf.read_file is deprecated. Please use tf.io.read_file instead.

WARNING:tensorflow:From ./DeepSpeech.py:375: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

W1010 05:29:49.134636 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:375: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From ./DeepSpeech.py:375: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
W1010 05:29:49.134779 140177034217280 deprecation.py:323] From ./DeepSpeech.py:375: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(dataset)`.
WARNING:tensorflow:From ./DeepSpeech.py:376: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
W1010 05:29:49.134902 140177034217280 deprecation.py:323] From ./DeepSpeech.py:376: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(dataset)`.
WARNING:tensorflow:From ./DeepSpeech.py:377: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
W1010 05:29:49.135006 140177034217280 deprecation.py:323] From ./DeepSpeech.py:377: DatasetV1.output_classes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(dataset)`.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W1010 05:29:49.137027 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W1010 05:29:49.137253 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W1010 05:29:49.137372 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
WARNING:tensorflow:From ./DeepSpeech.py:388: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W1010 05:29:49.216144 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:388: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From ./DeepSpeech.py:211: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W1010 05:29:49.219480 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:211: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1010 05:29:49.431784 140177034217280 deprecation.py:506] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From ./DeepSpeech.py:192: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

W1010 05:29:50.208915 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:192: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1010 05:29:50.260857 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From ./DeepSpeech.py:273: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1010 05:29:50.852802 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:273: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From ./DeepSpeech.py:332: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W1010 05:29:50.927818 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:332: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From ./DeepSpeech.py:410: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W1010 05:29:51.137559 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:410: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From ./DeepSpeech.py:414: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W1010 05:29:51.293362 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:414: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From ./DeepSpeech.py:416: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W1010 05:29:51.294395 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:416: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From ./DeepSpeech.py:421: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1010 05:29:51.299693 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:421: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W1010 05:29:51.413674 140177034217280 deprecation.py:323] From /home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/train-251862
I1010 05:29:51.430413 140177034217280 saver.py:1280] Restoring parameters from /home/jupyter/.local/share/deepspeech/checkpoints/train-251862
WARNING:tensorflow:From ./DeepSpeech.py:357: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

W1010 05:29:52.337543 140177034217280 deprecation_wrapper.py:119] From ./DeepSpeech.py:357: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.

I Restored variables from most recent checkpoint at /home/jupyter/.local/share/deepspeech/checkpoints/train-251862, step 251862
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:02 | Steps: 0 | Loss: 0.000000       
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000 | Dataset: /data/ldc93s1/ldc93s1.csv
Traceback (most recent call last):
  File "./DeepSpeech.py", line 832, in <module>
    tf.app.run(main)
  File "/home/jupyter/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/jupyter/.local/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/jupyter/.local/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "./DeepSpeech.py", line 816, in main
    train()
  File "./DeepSpeech.py", line 527, in train
    dev_loss = dev_loss / total_steps
ZeroDivisionError: float division by zero

Based on the info given in this discourse link, I checked my csv file too.

(nb_env) bash-4.2$ pwd
/data/ldc93s1
(nb_env) bash-4.2$ ls
ldc93s1.csv  LDC93S1.txt  LDC93S1.wav
(nb_env) bash-4.2$ cat ldc93s1.csv
wav_filename,wav_filesize,transcript
/data/ldc93s1/LDC93S1.wav,93638,she had your dark suit in greasy wash water all year
lissyx commented 4 years ago

I Restored variables from most recent checkpoint at /home/jupyter/.local/share/deepspeech/checkpoints/train-251862, step 251862

It looks like you have not cleaned up something as well. Please ensure you restart and verify from a clean state.

Also @MetaDhanesh I really need you to reproduce under a clean Python virtualenv and not under some Jupyter or anaconda binary. We have had errors due to that in the past.

MetaDhanesh commented 4 years ago

@lissyx I understand the need for clean python virtualenv. However, I am using GPU as shared resource which is accessible only through notebook. I am looking for a workaround whereby i can run it through clean python env.

Also, I changed the google dataset's file structure so that I can bring all of audio files into one directory. And since multiple files had same names under different sub-directories, I changed the file names. And hence the confusion.

Changed dataset link is here: google65k

lissyx commented 4 years ago

@lissyx I understand the need for clean python virtualenv. However, I am using GPU as shared resource which is accessible only through notebook. I am looking for a workaround whereby i can run it through clean python env.

Unless you reproduce under a proper clean env, I guess we'll be unable to help. It's 99.99% likely to be an issue on your system. Maybe because of CentOS, I don't know ...

lissyx commented 4 years ago

Also, I changed the google dataset's file structure so that I can bring all of audio files into one directory. And since multiple files had same names under different sub-directories, I changed the file names. And hence the confusion.

Changed dataset link is here: google65k

Ok, I will retry with those and let you know.

lissyx commented 4 years ago

@MetaDhanesh But in the meantime you can get a clean env, can you at least make sure you test without a stale checkpoint? Your last error is likely because of that ...

lissyx commented 4 years ago

However, I am using GPU as shared resource which is accessible only through notebook

If you can reproduce the issue with LDC93S1 dataset (please, ensure you don't have stale checkpoint), you don't need a GPU ...

lissyx commented 4 years ago

@MetaDhanesh So, I could train and export with your dataset:

INFO:tensorflow:Froze 12 variables.
I1011 12:08:48.784405 140341354239808 graph_util_impl.py:311] Froze 12 variables.
INFO:tensorflow:Converted 12 variables to const ops.
I1011 12:08:48.791962 140341354239808 graph_util_impl.py:364] Converted 12 variables to const ops.
I Exported model for TF Lite engine as output_graph.tflite
I Models exported at model_issue2414/
(tf-venv) alexandre@serveur:~/Documents/codaz/Mozilla/DeepSpeech/DeepSpeech-lissyx$ ll model_issue2414/
total 4,2M
drwxr-xr-x 1 alexandre alexandre   68 oct.  11 12:08 .
drwxr-xr-x 1 alexandre alexandre 2,3K oct.  11 12:02 ..
-rw-r--r-- 1 alexandre alexandre 3,3M oct.  11 12:07 output_graph.pb
-rw-r--r-- 1 alexandre alexandre 902K oct.  11 12:08 output_graph.tflite

This is definitively an issue on your side.

If you are required to use Jupyter for accessing the GPU, maybe you could:

If there's something badly interacting that is related to the Python/Jupyter setup, this could help.

lissyx commented 4 years ago

@MetaDhanesh Can you get us update ?

MetaDhanesh commented 4 years ago

I was able to run successfully on sample ldc93s1 dataset on virtual environment.

$ python3 DeepSpeech.py --n_hidden 256 --learning_rate 0.00001 --epochs 1 --export_tflite --export_dir data/modelgen_ldc93s1 --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv  --test_files data/ldc93s1/ldc93s1.csv

W1014 20:39:51.526147 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:494: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W1014 20:39:51.665138 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:348: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_types(iterator)`.
W1014 20:39:51.665330 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:349: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_shapes(iterator)`.
W1014 20:39:51.665493 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:351: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.data.get_output_classes(iterator)`.
W1014 20:39:53.206499 4551407040 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1014 20:39:53.208698 4551407040 deprecation.py:506] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1014 20:39:54.003494 4551407040 deprecation.py:323] From DeepSpeech.py:233: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1014 20:39:54.599544 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I1014 20:39:54.601692 4551407040 saver.py:1280] Restoring parameters from /Users/dhanesh/.local/share/deepspeech/checkpoints/train-2
I Restored variables from most recent checkpoint at /Users/dhanesh/.local/share/deepspeech/checkpoints/train-2, step 2
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 350.645477                                                                    
Epoch 0 | Validation | Elapsed Time: 0:00:00 | Steps: 1 | Loss: 348.510590 | Dataset: data/ldc93s1/ldc93s1.csv                                
I Saved new best validating model with loss 348.510590 to: /Users/dhanesh/.local/share/deepspeech/checkpoints/best_dev-3
I FINISHED optimization in 0:00:02.681611
I1014 20:39:57.697541 4551407040 saver.py:1280] Restoring parameters from /Users/dhanesh/.local/share/deepspeech/checkpoints/best_dev-3
I Restored variables from best validation checkpoint at /Users/dhanesh/.local/share/deepspeech/checkpoints/best_dev-3, step 3
Testing model on data/ldc93s1/ldc93s1.csv
Test epoch | Steps: 1 | Elapsed Time: 0:00:01                                                                                                 
Test on data/ldc93s1/ldc93s1.csv - WER: 1.000000, CER: 0.865385, loss: 348.510590
--------------------------------------------------------------------------------
WER: 1.000000, CER: 0.865385, loss: 348.510590
 - wav: file:///Users/dhanesh/learning/DeepSpeech/data/ldc93s1/LDC93S1.wav
 - src: "she had your dark suit in greasy wash water all year"
 - res: "cincinnati cachinnation cincinnati cincinnati"
--------------------------------------------------------------------------------
I Exporting the model...
W1014 20:39:59.277595 4551407040 deprecation_wrapper.py:119] From DeepSpeech.py:690: The name tf.nn.rnn_cell.LSTMStateTuple is deprecated. Please use tf.compat.v1.nn.rnn_cell.LSTMStateTuple instead.

W1014 20:39:59.316530 4551407040 deprecation.py:323] From DeepSpeech.py:129: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W1014 20:39:59.352936 4551407040 deprecation.py:323] From DeepSpeech.py:139: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell, unroll=True)`, which is equivalent to this API
W1014 20:39:59.365869 4551407040 deprecation.py:506] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py:961: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
I1014 20:40:00.327216 4551407040 saver.py:1280] Restoring parameters from /Users/dhanesh/.local/share/deepspeech/checkpoints/train-3
W1014 20:40:00.368676 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W1014 20:40:00.368858 4551407040 deprecation.py:323] From /Users/dhanesh/anaconda3/envs/machinelearning/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
I1014 20:40:00.399153 4551407040 graph_util_impl.py:311] Froze 12 variables.
I1014 20:40:00.417529 4551407040 graph_util_impl.py:364] Converted 12 variables to const ops.
I Exported model for TF Lite engine as output_graph.tflite
I Models exported at data/modelgen_ldc93s1

Now, I need to check on my GPU system, with a clean deepspeech setup. Thanks @lissyx

lissyx commented 4 years ago

Perfect, thanks for the update. So worst case you can train on GPUs and just export after. Make sure you keep in sync of versions. Can we close this?

MetaDhanesh commented 4 years ago

We can close this. Thanks for your help.

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.