Probabilistic adaptations of deterministic metrics

aimalz / proclam

PRObabilistic CLAssification Metrics for PLAsTiCC

Other

12 stars 7 forks source link

Probabilistic adaptations of deterministic metrics #78

Closed aimalz closed 5 years ago

aimalz commented 5 years ago

The referee requests an investigation of the F-score, among other deterministic metrics that can be extended to accommodate probabilistic classifications. This is our opportunity to finish the investigation that we started setting up for with #10 and #20.

At a minimum, we need to cover the well-trodden territory of AUC metrics:

[x] ROC AUC
[x] PRC AUC ~~- [ ] DET AUC~~ This one involves way too many arbitrary choices to do in an automated way.

We also need to look into broader ways to extend the following deterministic metrics:

[x] accuracy
[x] F-score
[x] Matthews correlation coefficient

Links to details about each can be found here.

aimalz commented 5 years ago

@djones1040, @mi-dai, and @aimalz are going to separate out the calculation of TP/FP/TN/FN into a single function in proclam.metrics.util.py and write a function for getting the AUC in proclam.metrics.util.py. Then proclam.metrics.roc.py, proclam.metrics.prc.py, and proclam.metrics.det.py can call those two utility functions.

aimalz commented 5 years ago

@mi-dai Could you add calls to roc.py in the notebook that runs the mock classifier tests for the paper? Thanks!