Passing arguments to a metric callable

[psorianom]

Hi !

First of all, thank you for this setfit implementation 😃

I want to pass the dict of metric arguments metric_kwargs to my metric callable but currently it is not possible: metric_kwargs is only passed to the evaluate package metrics:

        if isinstance(self.metric, str):
            metric_config = "multilabel" if self.model.multi_target_strategy is not None else None
            metric_fn = evaluate.load(self.metric, config_name=metric_config)
            metric_kwargs = self.metric_kwargs or {}

            return metric_fn.compute(predictions=y_pred, references=y_test, **metric_kwargs)

        elif callable(self.metric):
            return self.metric(y_pred, y_test)

https://github.com/huggingface/setfit/blob/cbc01ec402e86ca04e5e40e9bce7f618f3c2946c/src/setfit/trainer.py#L453C47-L453C47

The fix seems easy. Would it be something interesting to address ? Thank you !

[tomaarsen]

Hello!

This can definitely be addressed, I'll have a look at it shortly. In the meantime, you can pass those kwargs directly to your metric function :)

• Tom Aarsen

[psorianom]

Thank you for your quick reply.

Sorry, I was not clear, I am available to do this quick modification if needed :) Cheers,

[tomaarsen]

No need, I would like to include it in a large v1.0.0-pre branch that should be merged soon, rather than in main. As a result, it will be a bit easier if I take care of it :)

• Tom Aarsen

[psorianom]

Perfect. Thanks !