search

JuliaStats / MLBase.jl

A set of functions to support the development of machine learning algorithms
MIT License
185 stars 63 forks source link

Performance evaluation features #13

Open bkamins opened 9 years ago

bkamins commented 9 years ago

@lindahua, I would recommend adding the following features to Performance evaluation package:

High-level:

  1. Calculation of AUC (exact and approximated at subset of threshold values)
  2. Statistical comparison of difference between 2 ROC curves (pROC package from R is a good reference point)
  3. Bootstraped confitence intervals for calculated statistics
  4. Adding more functions that calculate statistics, those included in ROCR package from R are a good reference; the minimal required stats in my opinion would be rate of positive predictions, lift and cost (based on cost of fp an fn)
  5. Ability to calculate ROCNum not by threshold but by % of sample size (this is currently not available as it requires interpolation when there are several observations with exactly the same score and the % of sample is in the middle of their range; consider a vector of scores [1, 1, 1, 1, 2, 2, 2, 2] and asking about ROC at 10% of the sample - this problem is very typical for decision trees)

Low-level:

  1. Current interface is not very simple when applied for plotting (we get a vector of ROCNums), The functions operating on ROCNums (like false_negative etc.) could have methods that also work on Vector{ROCNums}
  2. It would be useful to store in ROCNums also threshold at which it was calculated (along with information about ord used)

I would recommend to discuss what you think that matches your idea of this package. I am also open to implement some of those additions.

jumutc commented 9 years ago

If anyone is interested I have a simple implementation of AUC based on ROCNum array which I embedded in my own project: SALSA.jl. I can introduce it very quickly to MLBase.jl.