search

mozilla / firefox-translations-training

Training pipelines for Firefox Translations neural machine translation models
https://mozilla.github.io/firefox-translations-training/
Mozilla Public License 2.0
143 stars 31 forks source link

Support training continuation for student models #515

Open gregtatum opened 5 months ago

gregtatum commented 5 months ago

We currently support training continuation for back translations, and teacher training.

eu9ene commented 4 months ago

This might be required to enable pre-emption for student models. cc @bhearsum

bhearsum commented 4 months ago

IIRC, this bug is about training continuation integrated with the training config.

I did just post a patch (#580) to automatically continue training for an earlier run in the same task though. The mechanism is a bit different, but I think that's the part we need to turn on spot instances again?

eu9ene commented 2 months ago

If we support continuation on preemption we can close this as we don't plan to use manual training continuation for the students now.