search

PaddlePaddle / Parakeet

PAddle PARAllel text-to-speech toolKIT (supporting Tacotron2, Transformer TTS, FastSpeech2/FastPitch, SpeedySpeech, WaveFlow and Parallel WaveGAN)
Other
598 stars 83 forks source link

WaveFlow training.. #102

Closed pavelxx1 closed 3 years ago

pavelxx1 commented 3 years ago

Hi, first -- thx for your repo) I have a question: Loss has negative values after 50 steps, its normal?? I use LJSpeech-1.1 dataset from repo example

https://prnt.sc/10w9knw part of training log:

Rank: 0, step: 3599, time: 0.000s/2.105s, loss: -4.310512
2021-03-26 15:42:42,302 - INFO - Rank: 0, step: 3599, time: 0.000s/2.105s, loss: -4.310512
Rank: 0, step: 3600, time: 0.000s/2.099s, loss: -3.645384
2021-03-26 15:42:44,404 - INFO - Rank: 0, step: 3600, time: 0.000s/2.099s, loss: -3.645384
Rank: 0, step: 3601, time: 0.000s/2.103s, loss: -3.686285
2021-03-26 15:42:46,511 - INFO - Rank: 0, step: 3601, time: 0.000s/2.103s, loss: -3.686285
Rank: 0, step: 3602, time: 0.000s/2.102s, loss: -4.348854
2021-03-26 15:42:48,616 - INFO - Rank: 0, step: 3602, time: 0.000s/2.102s, loss: -4.348854
Rank: 0, step: 3603, time: 0.000s/2.102s, loss: -3.750342
2021-03-26 15:42:50,721 - INFO - Rank: 0, step: 3603, time: 0.000s/2.102s, loss: -3.750342
Rank: 0, step: 3604, time: 0.001s/2.100s, loss: -3.806536
2021-03-26 15:42:52,825 - INFO - Rank: 0, step: 3604, time: 0.001s/2.100s, loss: -3.806536
Rank: 0, step: 3605, time: 0.000s/2.101s, loss: -4.202748
2021-03-26 15:42:54,928 - INFO - Rank: 0, step: 3605, time: 0.000s/2.101s, loss: -4.202748
Rank: 0, step: 3606, time: 0.000s/2.105s, loss: -4.336158
2021-03-26 15:42:57,036 - INFO - Rank: 0, step: 3606, time: 0.000s/2.105s, loss: -4.336158
Rank: 0, step: 3607, time: 0.000s/2.103s, loss: -4.162058
2021-03-26 15:42:59,142 - INFO - Rank: 0, step: 3607, time: 0.000s/2.103s, loss: -4.162058
Rank: 0, step: 3608, time: 0.000s/2.100s, loss: -3.776902
2021-03-26 15:43:01,245 - INFO - Rank: 0, step: 3608, time: 0.000s/2.100s, loss: -3.776902
Rank: 0, step: 3609, time: 0.000s/2.107s, loss: -3.573308
2021-03-26 15:43:03,355 - INFO - Rank: 0, step: 3609, time: 0.000s/2.107s, loss: -3.573308
Rank: 0, step: 3610, time: 0.000s/2.099s, loss: -4.175161
2021-03-26 15:43:05,457 - INFO - Rank: 0, step: 3610, time: 0.000s/2.099s, loss: -4.175161
Rank: 0, step: 3611, time: 0.000s/2.101s, loss: -3.867444
2021-03-26 15:43:07,561 - INFO - Rank: 0, step: 3611, time: 0.000s/2.101s, loss: -3.867444
[checkpoint] Saved model to /content/LJSpeech-1.1/TRAIN_DATA/_waveflow_out/checkpoints/step-3612.pdparams
[checkpoint] Saved optimzier state to /content/LJSpeech-1.1/TRAIN_DATA/_waveflow_out/checkpoints/step-3612.pdopt
iclementine commented 3 years ago

Yes. It is normal. The loss used for training waveflow is negative log probability density. Probability density could have a value larger than one, thus negative log probability density would have a negative value. In our previous experiment, the loss could go down to -4.7 to -5.2.