search

automl / auto-sklearn

Automated Machine Learning with scikit-learn
https://automl.github.io/auto-sklearn
BSD 3-Clause "New" or "Revised" License
7.65k stars 1.28k forks source link

Provide multi-class AUC ROC #1482

Open mfeurer opened 2 years ago

mfeurer commented 2 years ago

Scikit-learn provides multi-class options for area under curve: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

We should provide the most common ones, such as the OVO Macro averaging used by Auto-Gluon.

deadly-panda commented 2 years ago

Hey @mfeurer, can I try working on this ?

mfeurer commented 2 years ago

Sure, please go ahead. Please let us know if yo need further guidance.

deadly-panda commented 2 years ago

Just to make sure I'm on the right track. What I need to do is :

  1. Create the multi_class_roc_auc scorer (same as the auto-gluon link)
  2. Add unit tests for the scorer.
  3. Add the scorer to the classification metrics constant dictionary: CLASSIFICATION_METRICS
  4. In the 01_create_commands script.py: remove the check for nb class >2: continue

Is there something else I am missing?

mfeurer commented 2 years ago

Hey, I agree, except for the name. AutoGluon has a very specific name (ovo_macro) that describes the exact averaging mechanism, and it would be good if we could be as specific here, too. Also, you can have a look into our auto-sklearn code to see how we added these for f1, precision, etc.

deadly-panda commented 2 years ago

Hi @mfeurer .. Sorry it took too much time, I've been a bit busy. I have the code ready, however, I still have a few questions. For the unit tests, am not sure which numbers I should use ? (I created a PR, u can't take a look at the numbers I used). Also some tests were failling, for example TestMetric:test_classification_multiclass and I had to add the roc_auc metric to the list of skipped metrics, without really understanding why. can you give more details about this?

When I run black & mypy, there was some issues (from old code) with the common sub-module, should I also push those changes in the same PR.?

aseemk98 commented 1 year ago

Is this still open? I'd like to take a shot at this.

rohitsinghnaruka commented 4 months ago

is it still open ?? @mfeurer