search

NervanaSystems / deepspeech

DeepSpeech neon implementation
Apache License 2.0
222 stars 69 forks source link

"Problems restoring existing RNG state: algorithm must be 'MT19937' " - using weights from model zoo #4

Closed krishdeeplearning closed 7 years ago

krishdeeplearning commented 7 years ago

Hi,

I am getting the above warning when using the weights from the model zoo and running evaluate.py. Using the test-clean dataset, the model only seems to predict "QQQQQQQQ" for all the audio files. Is this because of the above warning? I am running on neon 1.7 in CPU mode.

Thanks!

tyler-nervana commented 7 years ago

Thanks for alerting us to this! I'll take a look.

tyler-nervana commented 7 years ago

The warning you see is unrelated. The issue is actually that there was a bug in the combination of RNNs + Clipped rectified linear activation + GPU. We use an optimized code path for that which is particularly fast, but erroneously omits the slope parameter of the activation function. Try adding this code to evaluate.py just after the model is loaded:

for ii, layer in enumerate(model.layers.layers):
    if isinstance(layer, BiBNRNN) and layer.activation.classnm == "Rectlinclip":
        layer.activation.slope = 0

We are submitting a patch to neon soon that will clean this up, but I just wanted to get you something quickly.

krishdeeplearning commented 7 years ago

The code fixed the issue for me! Thank you for quick response and fix!

tyler-nervana commented 7 years ago

That's great to hear. I'll leave this open until the fix is into neon.

tyler-nervana commented 7 years ago

This should now be fixed in neon 1.8.1.