search

rolczynski / Automatic-Speech-Recognition

🎧 Automatic Speech Recognition: DeepSpeech & Seq2Seq (TensorFlow)
GNU Affero General Public License v3.0
223 stars 64 forks source link

Skipping optimization due to error while loading function libraries #21

Closed michael-dm closed 4 years ago

michael-dm commented 4 years ago

Hello !

I run into this error while trying to make a simple prediction with this package : W tensorflow/core/grappler/optimizers/implementation_selector.cc:310] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference_standard_gru_13758' and '__inference_standard_gru_13758_specialized_for_bidirectional_3_2_backward_gru_3_StatefulPartitionedCall_at___inference_keras_scratch_graph_16375' both implement 'gru_fed7d4c6-fe30-487d-b5ea-3c3017a988bb' but their signatures do not match.

Any idea where that comes from ?

st-tomic commented 4 years ago

I am facing the same error when trying to run training. Finally the training loop brakes.

Seems like a Tensorflow issue, upgrading it to the latest version eliminates the error, but not sure if other functionality is affected.

rolczynski commented 4 years ago

Hey folks! I tried to simulate this error (new blank repo and the installation from pip), and all works fine. @st-tomic could you specify your suggestion about the TensorFlow upgrade? (from the error desc, it's related to the compatibility of the GRU implementation across different platforms - indeed, it was changing a little bit in the past).

st-tomic commented 4 years ago

Hi,

I have uninstalled Tensorflow 2.00 and just ran pip install tensorflow to get the latest stable version.

With latest tf it doesn't output this warning.

Switching back to suggested tf version (2.0 gpu) it outputs warning again:

2020-04-06 11:05:13.556498: W tensorflow/core/grappler/optimizers/implementation_selector.cc:310] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference___backward_cudnn_gru_with_fallback_1968_2109_specialized_for_training_Adam_gradients_gradients_bidirectional_2_backward_gru_2_StatefulPartitionedCall_grad_StatefulPartitionedCall_at___inference_keras_scratch_graph_14981' and '__inference___backward_standard_gru_10314_10873' both implement 'gru_1c338f59-abc5-4ea2-a56e-e3fcf2583724' but their signatures do not match.

rolczynski commented 4 years ago

The Tensorflow upgraded to 2.1.0