Closed valentinitnelav closed 2 years ago
The --nosave
might solve this. For job 3207748
, despite being terminated by the scheduler due to the set time limit of 50 hours, the weights folder had only 3 files and the model managed to run until epoch 300.
ls ~/PAI/detectors/yolov7/runs/train/yolov7_w6_b4_e300_hyp_custom/weights
# epoch_299.pt init.pt last.pt
Hi @stark-t
Do you have an idea from where to stop the train.py to save all these bunch of weights? This eats up a lot of storage pretty fast as the big ones are about 0.5 Gb. YOLOv5 saves only a best and last by default. I am confused about so many "best" options :D