Closed HiPatil closed 10 months ago
Hm seems to be something related to the pytorch dataloader.
Have never seen this before.
Perhaps you can set --cpu_cores 0
to turn of threading to see if that is the problem.
Does it happen immediately or just after a while of training?
It happens after a while of training. I will set cpu_cores 0
and let you know.
if it happens randomly after multiple epochs or so you can also just try to resume the training with --continue_epoch 1
and --load_file /path/to/latest_model.pth
I do observe that trainings can sometimes just crash for random reasons and need to be resumed. Compute clusters can be unstable.
Hi, I downloaded the dataset using the script provided and was trying to reproduce the results by training the model. While training, an error occurred, and I am not sure what is causing this. Would really appreciate the help.