search

mozilla / firefox-translations-training

Training pipelines for Firefox Translations neural machine translation models
https://mozilla.github.io/firefox-translations-training/
Mozilla Public License 2.0
145 stars 31 forks source link

Publish experiments to W&B from the CI #817

Closed vrigal closed 1 month ago

vrigal commented 1 month ago

Closes #792

vrigal commented 1 month ago

@eu9ene I found no simple way to disable cache in Taskcluster (usually I was updating training_config.marian-args.training-backward.disp-freq to trigger new jobs). Maybe it could be disabled directly from taskcluster ?

Otherwise any new pushes will trigger a CI job without training nor eval step, making it impossible to visualize on W&B.

eu9ene commented 1 month ago

@eu9ene I found no simple way to disable cache in Taskcluster (usually I was updating training_config.marian-args.training-backward.disp-freq to trigger new jobs). Maybe it could be disabled directly from taskcluster ?

Otherwise any new pushes will trigger a CI job without training nor eval step, making it impossible to visualize on W&B.

tracking is not included in cache digest by design since we don't want to rerun the core pipeline when tracking code changes. I think it's fine for now. It will be triggered by all the PRs that touch the training scripts and we can trigger it manually when we change tracking. We can ask @bhearsum whether it's possible to hack it and handle the latter case as well.