search

Rayhane-mamah / Tacotron-2

DeepMind's Tacotron-2 Tensorflow implementation
MIT License
2.27k stars 905 forks source link

During the inference, input the same data, the output will be different #363

Closed iamxiaoyubei closed 5 years ago

iamxiaoyubei commented 5 years ago

I used your code to train on another dataset. When training, it can get very good (eval) results. However, during the inference, I even could not synthesize a sound that could be heard. Input the same data, the output will be different (each time the sequence length has changed, and different from the input sequence length). I print "embedding input", "encoder output" and "decoder output" tensor to see what's wrong(in the following picture).

image

I found the problem occurred in the decoder. The same data passed through the decoder, the length and the value has changed. However, I can't take out the tensor inside of the decoder by any means. The following error will be reported.(I have tried many methods)

ValueError: Operation 'Tacotron_model/inference/decoder/while/CustomDecoderStep/decoder_prenet/dropout_2decoder_prenet/dropout/mul' has been marked as not fetchable.

Do you have any idea about my problems? Meanwhile, Would you share me your debugging tools? It is so hard to debug in the decoder. This problem has been bothering me for a week. Looking forward to your reply! Thank you very much!

alexdemartos commented 5 years ago

Hi,

this is due to dropout being applied at inference time. Dropout introduces randomness in the generation of the mel spectrograms.

iamxiaoyubei commented 5 years ago

@alexdemartos Thank you! It is indeed this problem. But after I fixed the dropout, the generated audio is still very bad. However, I think I have found the problem. It's very likely that my model is not well trained, but ground truth is used during the training phase, so it works well during the training phase, but it works poorly during the synthesis phase. Just like #206 . I think this model may not work for my dataset and input so far.

alexdemartos commented 5 years ago

The problem with this model is that the decoder RNN learns very well how to produce mels based on previous (ground-truth) outputs, but the alignment model is extremely weak. So basically your decoder doesn't care what the inputs are, it just learns that "after B goes C".

iamxiaoyubei commented 5 years ago

That's right. The good results in the evaluation once made me think that the model training is very good. However, I found that the alignment is very bad. I don't understand why I can't train a good alignment.