search

aimalz / proclam

PRObabilistic CLAssification Metrics for PLAsTiCC
Other
12 stars 7 forks source link

Exploratory metric implementation #5

Closed aimalz closed 6 years ago

aimalz commented 6 years ago

If you are interested in investigating a metric that is not already in the skeleton notebook, claim it here and do so.

aimalz commented 6 years ago

@tallamjr @cnsetzer Do you know what metrics you want to work on? I'm coordinating with @reneehlozek and @gnarayan on making mock data to run on but wanted to check that you were tasked for the interim before we give you mock data.

tallamjr commented 6 years ago

Thanks @aimalz . Unfortunately I do not think I will have time to work on this for a while, at the moment the main reason I have signed up is to stay in the loop with the on-going developments. However, I would be interested in learning more and playing with Matthews Correlation Coefficient (MCC) and G-mean metrics in the future.

cnsetzer commented 6 years ago

@aimalz I appreciate the check-in. I plan to spend the latter half of my day and when I can tomorrow looking at the different metrics in the notebook. I have had some unexpected time demands from my courses that has restricted my participation this week. I am quite new to ML, and the use of metrics in this context, so I would like to take a set of the metrics already implemented in scikitlearn (ROC, AUC, and Log-Loss to start) and use this opportunity to learn more about them and write the necessary notebook framework to pass through your mock data. I'll get to as many as I can.

aimalz commented 6 years ago

@tallamjr No problem! We'll try to keep this space updated, so you can jump in when you have time.

@cnsetzer Filling in the notebook with descriptions (mathematical definitions and implementation details) and minimal examples (even the ones in sklearn demos) is a great place to start! I'd recommend making a new branch for developing something based on the skeleton notebook and using this issue to ask questions and announce when you make significant pushes.

cnsetzer commented 6 years ago

@aimalz I have made my first commit of the metric implementations. I apologize I have not had much time until today to begin with these implementations but now I will be able to make progress with the multi-class versions of the ROC, AUC, and Log-loss which I claimed. I also plan to add more detail regarding these three metrics as discussed would be beneficial.

aimalz commented 6 years ago

@cnsetzer Your metric_implement notebook looks great! Since you mentioned getting started on multi-class metrics, something I forgot to include it in the original notebook that we should explore is modification of the class weights (i.e. not just in proportion to the number of class members) in the multi-class metrics and/or our own ways of combining the binary metrics into a final metric value, so maybe keep that in the back of your mind while you continue experimenting. Keep up the good work!

aimalz commented 6 years ago

The metrics are outgrowing this issue, so I'm going to start a new one for organizing something compatible with a pipeline for running all our metric tests.