ExpressAI / DataLab

The unified platform for data-related resources.
https://expressai.github.io/DataLab/
Apache License 2.0
131 stars 27 forks source link

Normalize WMT 20 metrics #420

Closed neubig closed 1 year ago

neubig commented 1 year ago

This PR normalizes WMT 20 metrics to be the same format as other NLG meta-evaluation datasets.

Example:

{'source': 'Brand in französischem Chemiewerk gelöscht', 'references': ['Fire extinguished in French chemical plant'], 'hypotheses': {'system_name': ['Online-G.1553', 'zlabs-nlp.1153', 'Online-A.1571', 'yolo.1052', 'Huoshan_Translate.789', 'UEDIN.1066', 'OPPO.1360', 'Online-B.1587', 'Tohoku-AIP-NTT.1442', 'WMTBiomedBaseline.387', 'Human-B.0', 'PROMT_NMT.77'], 'hypothesis': ['Fire extinguished in French chemical plant', 'Brand in French chemistry', 'Fire extinguished at French chemical plant', 'innovative legislation on democratic governance', 'Fire extinguished in French chemical plant', 'Fire extinguished in French chemical plant', 'Fire extinguished in French chemical plant', 'Fire extinguished in French chemical plant', 'Fire at French chemical plant extinguished', 'Fire extinguished in French chemical plant', 'Fire in French chemical plant extinguished', 'Fire extinguished in French chemical plant']}, 'scores': [99.0, 31.0, 99.5, 0.0, 100.0, 77.0, 100.0, 100.0, 100.0, 100.0, 76.0, 99.0], 'seg_id': 'salzburg.com.254713::1'}