search

GEM-benchmark / GEM-metrics

Automatic metrics for GEM tasks
https://gem-benchmark.com
MIT License
60 stars 20 forks source link

Fixes #76: Heavy metrics are not calculated in single-file mode #77

Closed madaan closed 2 years ago

madaan commented 2 years ago

This PR fixes #76.

In the single file mode, all the metrics are calculated serially. Thus the serial and parallel dictionaries are merged before calling compute. However, the dictionaries were being merged using:

serial_metric_dict.update(parallel_metric_dict)

which overwrites the heavy metrics originally present in serial_metric_dict. This patch fixes it by instead doing:

for metric_type, metric_list in parallel_metric_dict.items():
    if metric_type in serial_metric_dict:
        serial_metric_dict[metric_type].extend(metric_list)
    else:
        serial_metric_dict[metric_type] = metric_list