search

mozilla / firefox-translations-training

Training pipelines for Firefox Translations neural machine translation models
https://mozilla.github.io/firefox-translations-training/
Mozilla Public License 2.0
150 stars 32 forks source link

Consolidate all of the training scripts into a main pipeline/train/train.py script #607

Open gregtatum opened 4 months ago

gregtatum commented 4 months ago

We have:

pipeline/train/train.sh
taskcluster/scripts/pipeline/train-taskcluster.sh
taskcluster/scripts/pipeline/train_taskcluster.py

It will be much simpler to put this into a single training script. We should probably do this after our first big training run.

eu9ene commented 4 months ago

It's similar to #599 but I disagree that we should put Taskcluster-specific logic into the pipeline training script. The design so far was to make everything inside the pipeline directory independent of an orchestrator. It also makes it more testable. I think having one wrapper for Taskcluster specific needs is fine. We also already have #579. So if you agree on the current separation of concerns we can close this one.

gregtatum commented 4 weeks ago

I haven't commented on this one as I think it's worth looking at both approaches and come to a consensus on the design. In #842 I'm rewriting the train.sh to train.py, but the taskcluster bits were too complicated to initially take on.