About make_scorer

[kaiweiang]

Hi,

I wrote a custom scorer for sklearn.metrics.f1_score that overwrites the pos_label=1 by default and it looks like this

def custom_f1_score(y, y_pred, val):
   return sklearn.metrics.f1_score(y, y_pred, pos_label=val)

then I called this

scorer = make_scorer('f1_score', custom_f1_score, val=0)
automl.fit(X_train.copy(), y_train.copy(), metric=scorer, feat_type=feature_type)   

and I got this

Process pynisher function call:
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/root/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/lib/python3.6/site-packages/pynisher/limit_function_call.py", line 83, in subprocess_func
    return_value = ((func(*args, **kwargs), 0))
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/ensemble_builder.py", line 215, in main
    ensemble = self.fit_ensemble(selected_keys=selected_models)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/ensemble_builder.py", line 545, in fit_ensemble
    include_num_runs)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/ensembles/ensemble_selection.py", line 36, in fit
    self._fit(predictions, labels)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/ensembles/ensemble_selection.py", line 43, in _fit
    self._fast(predictions, labels)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/ensembles/ensemble_selection.py", line 91, in _fast
    all_scoring_functions=False)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/metrics/__init__.py", line 308, in calculate_score
    score = metric(solution, prediction)
  File "/root/anaconda3/lib/python3.6/site-packages/autosklearn/metrics/__init__.py", line 69, in __call__
    **self._kwargs)
TypeError: custom_f1_score() got multiple values for argument 'val'

I think I'm using the make_scorer wrongly, so please point me to the right direction. Thanks

[mfeurer]

Hey, I don't think you need that extra function around the actual score function. This snippet works on my side:

import sklearn.datasets

import autosklearn.classification
import autosklearn.metrics

def main():
    X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = \
        sklearn.model_selection.train_test_split(X, y, random_state=1)

    automl = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=120,
        per_run_time_limit=30,
    )

    scorer = autosklearn.metrics.make_scorer(
        'f1_score',
        sklearn.metrics.f1_score,
        pos_label=0,
    )
    automl.fit(X_train, y_train, dataset_name='digits', metric=scorer)

if __name__ == '__main__':
    main()

Please let me know if it does the job for you.

[kaiweiang]

@mfeurer yes it did. Thanks for the solution.

When I was running the automl.fit with metric=autosklearn.metrics.f1, I keep seeing this warning

/root/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.

whereas when I use the custom scorer from your solution above, I didn't see it. Why is that so?

Furthermore, If I can change the pos_label=0, this will solve the f1, precision, recall, and so. How about the sklearn.metrics.roc_auc_score, because it seems there is no pos_label parameter available?

Thank you

[mfeurer]

Yeah, that warning happens when the dummy prediction are scored, so as long as it happens only once it should be fine. Are the other results the way you expect them?

[kaiweiang]

So far I've only tested it with f1_score, but I think the scores such as precision and recall should work too when setting pos_label=0.

I've also tried on the roc_auc score. It seems like the score remains same regardless the positive class is 0 or 1. So is it safe to say the roc_auc is not affected when the positive class is set to zero, unlike f1?

[mfeurer]

So far I've only tested it with f1_score, but I think the scores such as precision and recall should work too when setting pos_label=0.

Most likely, I haven't tried though. Although it's obviously a different problem you're optimizing for.

I've also tried on the roc_auc score. It seems like the score remains same regardless the positive class is 0 or 1. So is it safe to say the roc_auc is not affected when the positive class is set to zero, unlike f1?

There is no argument pos_label for roc_auc_score in scikit-learn (see here)

[kaiweiang]

There is no argument pos_label for roc_auc_score in scikit-learn (see here)

Yes, I'm aware of that. I'm just not sure why it is not affected by the positive label whereas others like f1 or recall are. I guess I'll be asking this at stackoverflow.

Thanks

[mfeurer]

Closing this issue as the problem appears to be solved - please reopen if the problem still exists.