This repository contains data for the WMT 21 Metrics shared task Link to the paper: Results of the metrics shared task.
The German->English challenge set was created by Eleftherios Avramidis and Vivien Macketanz, and is distributed under a Creative Commons Attribution-ShareAlike 4.0 License.
Link to download all data (inputs + submissions) + code for computing correlations of metrics in the news, florestest and tedtalks domains
Link to download all inputs to the WMT 21 metrics task
Link to download only challenge set inputs for the WMT 21 metrics task
metric_submissions/all_inputs: Download all metric submissions for all inputs including newstest, florestest, tedtalks and challengesetsmetric_submissions/challenge_sets: Download all metric submissions for challenge sets only If you are using any of this data, please cite
@inproceedings{freitag-etal-2021-results,
title = "Results of the {WMT}21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on {TED} and News Domain",
author = "Freitag, Markus and
Rei, Ricardo and
Mathur, Nitika and
Lo, Chi-kiu and
Stewart, Craig and
Foster, George and
Lavie, Alon and
Bojar, Ond{\v{r}}ej",
booktitle = "Proceedings of the Sixth Conference on Machine Translation",
month = nov,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.wmt-1.73",
pages = "733--774"
}Note that Eleftherios Avramidis and Vivien Macketanz are working on a paper about the German->English challenge sets. We will add a citation to that paper when it is avalalable, and request that you also cite their paper if you are using data from the German->English challenge set.