Closed AshwinAce closed 5 years ago
Yes. Luminoth resumes from the last stored checkpoint. You might check this by watching the step number in the logs, during training.
Thanks! I stopped training way too soon, for the checkpoints to be saved. Based on the default settings, checkpoints are saved every 600 seconds.
The command for training on a dataset is lumi train -c path_to_config.yml For resuming this on the cloud, we have optional commands which feed the job-id for the same. How do we resume the same when run on a local workstation? Does it automatically resume from the previous trained checkpoint? Currently, I am terminating the program as runtime is very high. I want to verify that resume works even when the termination is forced as such.