Open gregtatum opened 5 months ago
This might be required to enable pre-emption for student models. cc @bhearsum
IIRC, this bug is about training continuation integrated with the training config.
I did just post a patch (#580) to automatically continue training for an earlier run in the same task though. The mechanism is a bit different, but I think that's the part we need to turn on spot instances again?
If we support continuation on preemption we can close this as we don't plan to use manual training continuation for the students now.
We currently support training continuation for back translations, and teacher training.