Closed kaiweiang closed 5 years ago
Hey, I don't think you need that extra function around the actual score function. This snippet works on my side:
import sklearn.datasets
import autosklearn.classification
import autosklearn.metrics
def main():
X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = \
sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier(
time_left_for_this_task=120,
per_run_time_limit=30,
)
scorer = autosklearn.metrics.make_scorer(
'f1_score',
sklearn.metrics.f1_score,
pos_label=0,
)
automl.fit(X_train, y_train, dataset_name='digits', metric=scorer)
if __name__ == '__main__':
main()
Please let me know if it does the job for you.
@mfeurer yes it did. Thanks for the solution.
When I was running the automl.fit with metric=autosklearn.metrics.f1
, I keep seeing this warning
/root/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
whereas when I use the custom scorer from your solution above, I didn't see it. Why is that so?
Furthermore, If I can change the pos_label=0
, this will solve the f1, precision, recall, and so. How about the sklearn.metrics.roc_auc_score
, because it seems there is no pos_label
parameter available?
Thank you
Yeah, that warning happens when the dummy prediction are scored, so as long as it happens only once it should be fine. Are the other results the way you expect them?
So far I've only tested it with f1_score, but I think the scores such as precision and recall should work too when setting pos_label=0
.
I've also tried on the roc_auc
score. It seems like the score remains same regardless the positive class is 0 or 1. So is it safe to say the roc_auc
is not affected when the positive class is set to zero, unlike f1
?
So far I've only tested it with f1_score, but I think the scores such as precision and recall should work too when setting pos_label=0.
Most likely, I haven't tried though. Although it's obviously a different problem you're optimizing for.
I've also tried on the roc_auc score. It seems like the score remains same regardless the positive class is 0 or 1. So is it safe to say the roc_auc is not affected when the positive class is set to zero, unlike f1?
There is no argument pos_label
for roc_auc_score
in scikit-learn (see here)
There is no argument pos_label for roc_auc_score in scikit-learn (see here)
Yes, I'm aware of that. I'm just not sure why it is not affected by the positive label whereas others like f1 or recall are. I guess I'll be asking this at stackoverflow.
Thanks
Closing this issue as the problem appears to be solved - please reopen if the problem still exists.
Hi,
I wrote a custom scorer for
sklearn.metrics.f1_score
that overwrites thepos_label=1
by default and it looks like thisthen I called this
and I got this
I think I'm using the make_scorer wrongly, so please point me to the right direction. Thanks