Open WGierke opened 6 years ago
For the both identification and classification (false positive reduction) tasks was proposed a handy evaluation framework by the LUNA16 authors. They employed Free-Response Receiver Operating Characteristic (FROC) and competition performance metric (CPM). It computes an average of the seven sensitivities measured at several false positives per scan (FPPS) thresholds, more concretely, at each FPPS ∈ {0.125, 0.25, 0.5, 1, 2, 4, 8} true positive rate was computed. Mean of which forms the CPM. From my point of view, it worth to pay attention to the CPM neither the logloss. I can work on that to adjust their pipeline, if no one mind.
@vessemer great observation! I see that they provide evaluation code as well. So yes, adjusting it to fit our use case will be extremely useful!
As discussed in #131 it would be helpful to have a consistent pipeline to evaluate prediction models. This way we get to know how well the currently implemented models are, which ones need to be improved and how well a new model performs. The pipeline should calculate the appropriate metrics that have been specified in #221 while some of the are already available here.
Acceptance Criteria
Please refer to the PR template for further explanations of the metrics.