Improve automatic quality evaluation

[eu9ene]

This is a meta issue to brainstorm ideas on how to make our final automatic quality evaluation procedure more robust and suitable for decision-making. This currently happens in firefox-translations-models repo but will be migrated to the main repo and W&B.

See this doc.

Some ideas:

• Consider reporting absolute difference instead of %
• Consider adding more metrics recommended in the literature
• Consider adding more interpretable metrics where absolute values indicate readiness for release, for example, xCOMET
• Try not to use validation datasets where possible (remove flores-dev)
• Inspect how representative the datasets are compared to what users translate in the browser
• Use the same datasets for all languages as a final metric ( either only Flores or build a new dataset)

Recommendations from the paper Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies:

• Use CometKiwiQE22 as the main metric. In addition to its better performance, as a quality estimation metric, it is not affected by references.
• Use at least one additional metric of a different type; e. g. BLEURT20, which is reference-based and uses a different architecture from Comet.
• For each metric delta, report estimated accuracy to help align reliability of used metrics.
• Do not use BLEU, ChrF, or spBLEU to evaluate unrelated systems.

We can add a task list here after migrating [firefox-translations-models].

[marco-c]

See also #229.