IBM / unitxt

🦄 Unitxt: a python library for getting data fired up and set for training and evaluation
https://unitxt.rtfd.io
Apache License 2.0
139 stars 29 forks source link

Fix MRR wiring #958

Closed assaftibm closed 4 days ago

assaftibm commented 1 week ago

The underlying implementation has changed, and now it requires the reference to be provided as List[List[str]] rather than just List[str]. We want to expose the metric in its standard definition, where the reference is List[str]. @benjaminsznajder noticed this and can give more details.

matanor commented 5 days ago

Suggested fix from @yoavkatz :

 from unitxt.collections_operators import Wrap
metric = MetricPipeline(
        main_score=“score”,
        preprocess_steps=[
            Copy(field=“context_ids”, to_field=“prediction”),
            Wrap(field=“ground_truths_context_ids”, inside=“list”, to_field=“references”),
        ],
        metric=f”metrics.{metric_name}“,
    )