Compute the standard deviation of COMET scores for training student models

mozilla / translations

The code, training pipeline, and models that power Firefox Translations

https://mozilla.github.io/translations/

Mozilla Public License 2.0

154 stars 33 forks source link

Compute the standard deviation of COMET scores for training student models #885

Closed gregtatum closed 2 weeks ago

gregtatum commented 2 weeks ago

In #771 I had a bug in my code, where the training data wasn't actually truncated as expected, and so the models were trained with identical data. The lucky benefit of this was that we can compute a sample standard deviation of scores from that.

The result was: +/- 00.12 difference.

Type	Value
COMET	89.48
COMET	89.24
COMET	89.41
COMET	89.50
STDDEV	00.12