Interrupting and resuming training from last epoch

[EmanueleGiavardi]

❓ Questions

Hi! Is there a way to safely interrupt training at a specific epoch and then, after some time, resume the same experiment from the point where it was paused? In the documentation, I noticed that dora run -d -f [SIG] continue_from=[SIG] can be used to initiate a new experiment using hyperparameters and weights from the specified experiment... but I would like to resume training from the exact epoch where it was interrupted, rather than starting a new one from scratch.

[CarlGao4]

As long as you didn't change your variant, you can just run the same command as the first time you train

e.g. You run dora run variant=my_variant first, then interpreted after a few epoches. Next time you can just run dora run variant=my_variant again and training will continue from the last epoch which has been trained completely you stopped.

[EmanueleGiavardi]

As long as you didn't change your variant, you can just run the same command as the first time you train

e.g. You run dora run variant=my_variant first, then interpreted after a few epoches. Next time you can just run dora run variant=my_variant again and training will continue from the last epoch which has been trained completely you stopped.

I have attempted this multiple times, but I am confused by the fact that, when attempting to resume training from where it was previously interrupted, the logs indicate that the training restarts from epoch 1 instead of picking up from the last epoch before the interruption

[CarlGao4]

Did you modify the codes? Please also check outputs/xps/[xps]/history.json.

[EmanueleGiavardi]

Did you modify the codes? Please also check outputs/xps/[xps]/history.json.

mhh I don't have history.json in the xp folder. I made some changes to the code, but I don't recall making any modifications related to the writing of that file. I'll check the modifications that I've made

[CarlGao4]

You should keep this line: https://github.com/facebookresearch/demucs/blob/e976d93ecc3865e5757426930257e200846a520a/demucs/solver.py#L282

[EmanueleGiavardi]

You should keep this line:

https://github.com/facebookresearch/demucs/blob/e976d93ecc3865e5757426930257e200846a520a/demucs/solver.py#L282

Right! I commented that section of the code to avoid old sdr calculation at the end of the training, but I left self.link.push_metric(metrics) commented by mistake. Thank you!!