Closed dillfrescott closed 1 year ago
Happened twice now at different spots
Hi, I'm not sure how to help you debug this as you are working on a different dataset and configuration. It looks like you have only 23 training samples with about 30 seconds per utterance.
The error message suggests that there is something wrong when processing input data. It may also be some bug that only happens when you have small amount of data.
I suspect the sampler may be an issue. Since you have only 23 samples, bucketing no longer helps and may be an issue. You can try replacing https://github.com/b04901014/UUVC/blob/master/trainer.py#L33 by something like:
def train_dataloader(self):
dataset = data.DataLoader(self.traindata,
num_workers=self.hp.nworkers,
batch_size=1,
collate_fn=self.traindata.seqCollate)
return dataset
to disable bucketing, and see if that subdues the error.
Thank you so much! One other question:
How would I resume training from a ckpt?
You can do so by passing with the argument of --resume_checkpoint
or --pretrained_path
.
pretrained_path
only loads the model weights and trains from scratch, and resume_checkpoint
restores all the training states (e.g., optimizer parameters, number of epochs)
Thank you so much again! I'm gonna close this issue because the code you provided seems to have fixed the issue (I'm at 15 epochs so far and it hasn't errored out)