search

GEM-benchmark / GEM-metrics

Automatic metrics for GEM tasks
https://gem-benchmark.com
MIT License
60 stars 20 forks source link

Modified fix for `references` vs. `targets`. #98

Closed tuetschek closed 2 years ago

tuetschek commented 2 years ago

This is a slightly more general solution for #97. The original code would only look for the "references" key if the data was loaded from a file, it would've had no effect if you passed a pre-loaded JSON. Also, the "references" key had priority regardless of the subclass of Texts, so if you wanted to load references, sources and predictions from the same file (with a list of instances with 3 keys), it would always load for references.

This version delegates the decision on data_key back to subclasses, only References now prioritize "references" over "targets". I also added some docs for the base class to point this out.

jordiclive commented 2 years ago

@tuetschek nice! Just tested and the results are as expected. This is indeed a lot better.