search

IBM / unitxt

🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
https://www.unitxt.ai
Apache License 2.0
151 stars 38 forks source link

Support disambiguation of scores from metrics that returns same score names #802

Open yoavkatz opened 4 months ago

yoavkatz commented 4 months ago

Today, If metrics, returns the same score (e.g. "f1") then the first metric score is returned, and the second metric is ignored.

metrics ["metrics.bert_score.distilbert_base_uncased","metrics.bert_score.deberta_base_mnli"],

It would be good to be able to add a prefix:

metrics ["metrics.bert_score.distilbert_base_uncased[score_prefix="distilbert_]","metrics.bert_score.deberta_base_mnli[score_prefix="deberta_]"],

The score prefix will be appended to any score returned by the metrics (except score_name, and score which are reserved names).

elronbandel commented 4 months ago

I think @arielge also dealt with it. @arielge if you have more to add to this issue feel free.

arielge commented 4 months ago

Indeed, I encountered this issue in the context of overwrite args - https://github.com/IBM/unitxt/issues/737, but nothing to add otherwise