Open ghost opened 2 years ago
Hi, when I train the model with the script run_dist.sh, the checkpoint can not be saved when the first epoch finishes. However, the gpu is still occupied and there is no more logs. Any advise ?
Hi, when I train the model with the script run_dist.sh, the checkpoint can not be saved when the first epoch finishes. However, the gpu is still occupied and there is no more logs. Any advise ?