search

occam-ra / occam

OCCAM Reconstructability Analysis Tools
Other
20 stars 14 forks source link

Output confusion matrix and ROC curve #27

Open gdcutting opened 5 years ago

gdcutting commented 5 years ago

"Directed, Fit, Test data included: Always output confusion matrix and ROC curve. Explanation & Algorithm: Right now, if |DV|>2, the user has the option of filling in a box with the state to be used as the "negative," i.e., good, state; all other DV states will then be aggregated for purposes of confusion matrix output. If the user doesn't specify anything in this box, a confusion matrix is now not generated. Change this to the following. If the user specifies the "negative" state, use it, but if the user doesn't specify a "negative" state, automatically chose the DV state that has the highest probability in the marginal distribution, and output the state that has been chose. Thus when test data is included, Fit should always output a confusion matrix. Fit should also output an ROC curve. Right now, Fit outputs the prediction rule determined by whichever q(DV_i|IV) is the biggest for every IV state. This amounts to predicting the "negative" state if its calculated conditional probability is greater than 0.5; otherwise Fit predicts the other, "positive" (bad), state. An ROC curve is generated by changing this 0.5 cutoff to vary from 0 to 1 in small steps."

venkatachalapathy commented 5 years ago

I found a standard R package for ROC curve. Maybe this is useful for you, @gdcutting

https://cran.r-project.org/web/packages/ROCR/index.html