I don't if this is related to me deleting previous models in the folder, but when training for a while each day, someday it just stop to continue from last epoch and started from epoch 1 again, even if last Discriminator and G model was saved correctly (I guess).
I didn't deleted D and G, only small models because there weren't good yet and they were occupying large space in my SSD.
train with this settings stopping and continuing train for a few times, and try to delete small models that are not the best model and not the last model each time you stop training, keeping D and G models.
like this:
D_233333333.pth
G_233333333.pth
name_10e_step_best_epoch.pth
name_10e_step.pth
Project Version
3.2.3
Platform and OS Version
Linux x64
Affected Devices
PC
Existing Issues
No response
What happened?
I don't if this is related to me deleting previous models in the folder, but when training for a while each day, someday it just stop to continue from last epoch and started from epoch 1 again, even if last Discriminator and G model was saved correctly (I guess). I didn't deleted D and G, only small models because there weren't good yet and they were occupying large space in my SSD.
Steps to reproduce
sample_rate = '48000' batch_size = '6' total_epochs = '200' #42 save_every_epoch = '1' save_only_latest = True save_every_weights = True custom_pretrained = True overtraining_detector=True overtraining_threshold=15 d_path = 'pretraineds/D-f048k-TITAN-Medium.pth' g_path = 'pretraineds/G-f048k-TITAN-Medium.pth'
2.
train with this settings stopping and continuing train for a few times, and try to delete small models that are not the best model and not the last model each time you stop training, keeping D and G models. like this: D_233333333.pth G_233333333.pth name_10e_step_best_epoch.pth name_10e_step.pth
Expected behavior
continue training from last epoch done
Attachments
No response
Screenshots or Videos
No response
Additional Information
No response