This is a crude framework we used and is described in https://github.com/mozilla/firefox-translations/issues/231#issuecomment-1092250963. Please feel free to use it as a reference or a starting point. Essentially this checks that translating 1M sentences happen within some desired time limit and is very close to the efficiency-task. Using similar parameters will provide comparability to other technologies amongst participants in the efficiency-task.
Unfortunately Security audits have shut our Jenkins down (deeming it unnecessary) and bringing it back up is not a hill anyone in statmt wants to die on. We are YOLOing speed-tests at the moment so appreciate if this gets solved here at some point.
https://github.com/browsermt/bergamot-translator-tests/blob/main/speed-tests/test_wngt20_perf.sh
This is a crude framework we used and is described in https://github.com/mozilla/firefox-translations/issues/231#issuecomment-1092250963. Please feel free to use it as a reference or a starting point. Essentially this checks that translating 1M sentences happen within some desired time limit and is very close to the efficiency-task. Using similar parameters will provide comparability to other technologies amongst participants in the efficiency-task.
Unfortunately Security audits have shut our Jenkins down (deeming it unnecessary) and bringing it back up is not a hill anyone in statmt wants to die on. We are YOLOing speed-tests at the moment so appreciate if this gets solved here at some point.