search

GEM-benchmark / GEM-metrics

Automatic metrics for GEM tasks
https://gem-benchmark.com
MIT License
60 stars 20 forks source link

Adding Two Diversity Metrics: TTR and Yule's I-Measure #86

Closed vyraun closed 2 years ago

vyraun commented 2 years ago

This PR adds two new metrics to GEM: TTR and Yule's I-Measure, as described in "Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation", Vanmassenhove et al., EACL 2021.

  1. TTR (Type to Token Ratio) presents the ratio of the total number of different words (types) to the total number of words (tokens). Higher TTR indicates a higher degree of lexical diversity.
  2. Yule’s characteristic constant (Yule’s K) measures constancy of text as the repetitiveness of vocabulary. Yule’s K and its inverse Yule’s I measure (implemented here) are considered to be more resilient to fluctuations related to text length than TTR.

Please let me know if any changes would be helpful prior to the integration. Thanks.

tuetschek commented 2 years ago

Hy @vyraun , thanks for the PR! The code looks good to me. Could you please add tests for both metrics before we merge it?

You can have a look at the instructions at https://github.com/GEM-benchmark/GEM-metrics/tree/main/tests & the files in this directory to see how the tests work – it shouldn't be hard to add them.

vyraun commented 2 years ago

Thanks @tuetschek ! Sorry for the late reply, I have updated the PR with the tests for both metrics. Please let me know if any other changes would be useful. Thanks.