mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
155 stars 34 forks source link

Test WASM Translations in CI #927

Closed nordzilla closed 1 week ago

nordzilla commented 1 week ago

This PR adds functionality to test our generated JavaScript WASM translations end-to-end in CI, as well as adding an eslint config/task for the new JS files.

This is a fairly large changeset, but each commit is broken out into a purposeful set of changes. I recommend reviewing commit by commit.


@mozilla/releng @bhearsum I think the only directly related commit for you is Add new lint tasks for eslint, which modifies the relevant kind.yml file.


@gregtatum all of the new functionality can be run/tested locally with the following commands:

task lint-eslint
task lint-eslint-fix
task lint
task lint-fix
task docker-run -- task inference-clean
task docker-run -- task inference-test-wasm
nordzilla commented 1 week ago

@bhearsum @gregtatum

I'm going to have to re-request your reviews here after applying review feedback.


For @bhearsum

I've added a new commit that affects taskcluster docker images:

It made the most sense to me to modify the base docker image, since the configuration for git-lfs is at the project root in the .gitattributes file.

There is a Matrix discussion thread that you can read about my approach here.

I also have a question for you in this comment.


For @gregtatum

Add a subset of models for testing using git-lfs now adds only the git-lfs definitions instead of the whole model archive.

Extract test models from archives in test-wasm.py now pulls the files via git-lfs before extracting them from their archives.

This ensures that we're not checking the large model files into tree.

nordzilla commented 1 week ago

Okay @bhearsum, I believe I've fixed all of the issues with the docker known_hosts.

The primary commit is still Add support for git-lfs to base docker image