search

mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

Rework wasm build scripts for gecko #902

Closed nordzilla closed 1 day ago

nordzilla commented 1 week ago

@gregtatum

The goal of this patch is to move much of the functionality from the build-bergamot.py script in Mozilla Central upstream into this repository to better streamline how WASM artifacts are built and exported for use in Gecko.

There are three major changes that occur:

1) The current version of build-bergamot.py applies a patch to the upstream Bergamot source code after cloning the repository. This is no longer necessary because we can make the change permanently upstream here.

2) build-bergamot.py is moved in its entirety into this repository as build-wasm.py. It is then modified to work within this ecosystem. I moved the whole file first so that you can look commit by commit to clearly see what changed from the original source.

3) The docker-run task now utilizes Python instead of bash, and it now has the ability for the user to specify additional volumes to be shared between the host file system and the docker container. 

task docker-run -- task inference-build-wasm --volume "HOST_PATH/build-wasm:inference/build-wasm"

This allows us to build the WASM artifacts within docker and export them directly into tree when making updates in Mozilla Central.

I will also be submitting a sister patch on the Mozilla Central side that updates the fetch/build scripts to utilize this new functionality.

nordzilla commented 4 days ago

I vaguely recall having this same issue with the original build-bergamot.py script, because the part where the git patches are applied happens before calling cmake, and cmake is the command that ensures all the submodules are availalbe. It works if you've already pulled the submodules, but may not work the first time.

I'm going to try reordering those steps.

Hopefully this fixes it.

nordzilla commented 3 days ago

@gregtatum

I've pushed up two new commits.

Firstly, I fixed the submodule initialization to ensure that the directories exist before applying the patches. I've tested this end-to-end on my macOS computer both in this repo and in mozilla-unified.

Secondly, my macOS machine seems to fail the build if I pass -j N to make where N > 1, so I've added the ability to pass -j to the build script as well. The default is to use the number of logical cores on the CPU, so as not to slow down CI.

I've also changed the in-tree script in D226939 to pass -j 1 when building in tree in Firefox. I figure that we do that so little that it wouldn't hurt to build on a single thread when publishing.

One could always modify that file temporarily if they're iterating on it locally in tree.