search

Rayhane-mamah / Tacotron-2

DeepMind's Tacotron-2 Tensorflow implementation
MIT License
2.27k stars 905 forks source link

Loss exploded on some datasets #345

Closed XiaooaiX closed 5 years ago

XiaooaiX commented 5 years ago

Dear all, I have a problem these days about this repo that when I train this taco2 model on some dataset, the loss appears to get exploded after a number of steps. Just like this:

capture

Anyone knows how to fix it?

Thanks!

Bobb10 commented 5 years ago

@XiaooaiX. Can you share the solution for this issue? I have the same problem with a custom dataset, its loss explodes after 35K of wavenet training steps.

dalvlv commented 5 years ago

@XiaooaiX Hi friend, did solve the 'Loss Exploded' problem? When I trained Mandarin dataset, I met the same problem. How did you solve it?

dalvlv commented 5 years ago

@Bobb10. Loss value of my train turns to negatigve quickly at start, do you know the reason？ And also after 40K steps, the loss exploded.

Bobb10 commented 5 years ago

@dalvlv. I haven't found a reason why the loss explodes while training WaveNet. It explodes sort of randomly after several steps (not just 35K, but also further in the training stage). Would be more than happy to receive an answer if someone knows why this is happening and how to solve this.