Closed jjmachan closed 6 days ago
Let's say I am evaluating using an evaluation dataset that contains user_input, response, reference, and retrieved_context. Consider scenarios of the user using two metrics with this dataset, ie context_recall and aspect critic (as harmfulness or something) at the same time. Using this interface, even if my aspect critic metric does not need retrieved context it will use it. There is no way to opt-out. I think this can occur when using the metrics using the evaluate
interface.
When using a single metric this seems to be fine. @jjmachan
Metrics Converted
a few different examples
Aspect Critic
with reference
Note: this only works for multi-turn metrics for now