optuna / optuna-examples

Examples for https://github.com/optuna/optuna
MIT License
692 stars 178 forks source link

Use auc score to unify the intermediate and objective values #272

Closed nzw0301 closed 4 months ago

nzw0301 commented 4 months ago

Motivation

Resolve https://github.com/optuna/optuna-examples/issues/251.

Description of the changes

Use pre-computed booster's validation auc score rather than computing inconsistent accuracy.

Alternative ways I considered were not working by the following reasons:

y0z commented 4 months ago

@not522 @eukaryo Could you review this PR?

not522 commented 4 months ago

Sorry for my late response. How about using sklearn.metrics.roc_auc_score(valid_y, preds)? It seems like a more usual way of calculating AUC. (I'm not familiar with LightGBM, so please correct me if I'm wrong.)

nzw0301 commented 4 months ago

Thank you for your suggestion! You're absolutely right and my comment:

Compute AUC score using sklearn: We cannot calculate the AUC score because gbm does not have predict_prob method https://github.com/dmlc/xgboost/issues/4583, which is necessary to compute the score

was wrong. I'll update the file by following your suggestion.