search

p0p4k / vits2_pytorch

unofficial vits2-TTS implementation in pytorch
https://arxiv.org/abs/2307.16430
MIT License
465 stars 81 forks source link

After finishing the training, continuing the training revealed that the curve of learning rate had reversed。 #88

Open Mr47121836 opened 2 months ago

Mr47121836 commented 2 months ago

image image This is my log and tensorboard log.

p0p4k commented 2 months ago

Did you stop and continue training?

Mr47121836 commented 2 months ago

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。

Mr47121836 commented 2 months ago

Did you stop and continue training?

yes，when i stop and continue training，i discover learnning rate is not the last learnning rate.And I also discover orignal vits also have this situation. This is vits train image

p0p4k commented 2 months ago

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

Mr47121836 commented 2 months ago

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

but,When I continued training, I found a huge change in the curve. image image image I thing it can't load the checkpoint ?

Mr47121836 commented 2 months ago

Yea its normal, the checkpoint saved at different step and continue is from that step, while model was trained a little bit more steps in the first run, the board shows everything. You can sort using Wall button in the tensorboard in the left side.

but,When I continued training, I found a huge change in the curve. image image image I thing it can't load the checkpoint ?

image The total loss have a huge change