Training time limitation

[DudusEscariot]

Managed to get the training colab notebook to work! Was battling with the code for over 24 hrs.

For some reason the training will only go up to "Epoch: 4 Iter: 2500" (15 mins or so) then it stops, and I would like to train it for longer if possible, is there some setting I'm missing?

[DudusEscariot]

2022-10-10 09:30:48,533 INFO: [001_E..][epoch: 4, iter: 2,500, lr:(1.000e-04,)] [eta: 13:27:46, time (data): 0.327 (0.003)] l_g_pix: 2.5152e-04 l_g_percep: 6.4053e-01 l_g_gan: 1.8789e-02 l_d_real: 4.2093e-02 l_d_fake: 2.2214e-02 out_d_real: -1.3722e+01 out_d_fake: -1.7415e+01 2022-10-10 09:30:48,534 INFO: Saving models and training states. Traceback (most recent call last): File "train.py", line 221, in train_pipeline(root_path) File "train.py", line 200, in train_pipeline model.validation(val_loader, current_iter, tb_logger, opt['val']['save_img']) File "/content/wav2lip-hq/basicsr/models/base_model.py", line 49, in validation self.nondist_validation(dataloader, current_iter, tb_logger, save_img) File "/content/wav2lip-hq/basicsr/models/sr_model.py", line 136, in nondist_validation dataset_name = dataloader.dataset.opt['name'] AttributeError: 'NoneType' object has no attribute 'dataset'

[DudusEscariot]

SOLVED...

In train_basicsr changed (changes in BOLD):

validation settings

val: val_freq: !!float 100e2 save_img: true

metrics: psnr: # metric name, can be arbitrary type: calculate_psnr crop_border: 4 test_y_channel: false

logging settings

logger: print_freq: 100 save_checkpoint_freq: !!float 100e2