search

theAIGuysCode / YOLOv4-Cloud-Tutorial

This repository walks you through how to Build and Run YOLOv4 Object Detections with Darknet in the Cloud with Google Colab.
355 stars 300 forks source link

Colab disconnects and all the files in the colab VM are lost #4

Open rishabh700 opened 4 years ago

rishabh700 commented 4 years ago

In case when the colab gets disconnected in the middle of the training, all the files in the VM are lost (such as the darknet folder). Does this mean that I will have to clone the darknet repo and upload all the other pre-requisite files (images, labels, etc) again to start the training?
If yes, would it be helpful to also backup the darknet folder with the gdrive (in addition to the weights file)?

adhilcodes commented 4 years ago

1 ) Yes you have to clone the darknet repo and upload all the other pre-requisite files (images, labels, etc) again to start the training once it is terminated from the training, the storage in ColabVM is temporary .

2) After every 100 iterations your weights are saved in the backup folder in your Gdrive , Also after every 1000 iterations also your weights get saved in the backup folder. So if you terminate from the training you can restart in from where did you stopped it.