drivendataorg / concept-to-clinic

ALCF Concept to Clinic Challenge
https://concepttoclinic.drivendata.org/
MIT License
367 stars 147 forks source link

Evaluation Pipeline for Models #271

Open WGierke opened 6 years ago

WGierke commented 6 years ago

As discussed in #131 it would be helpful to have a consistent pipeline to evaluate prediction models. This way we get to know how well the currently implemented models are, which ones need to be improved and how well a new model performs. The pipeline should calculate the appropriate metrics that have been specified in #221 while some of the are already available here.

Acceptance Criteria

Please refer to the PR template for further explanations of the metrics.

vessemer commented 6 years ago

For the both identification and classification (false positive reduction) tasks was proposed a handy evaluation framework by the LUNA16 authors. They employed Free-Response Receiver Operating Characteristic (FROC) and competition performance metric (CPM). It computes an average of the seven sensitivities measured at several false positives per scan (FPPS) thresholds, more concretely, at each FPPS ∈ {0.125, 0.25, 0.5, 1, 2, 4, 8} true positive rate was computed. Mean of which forms the CPM. From my point of view, it worth to pay attention to the CPM neither the logloss. I can work on that to adjust their pipeline, if no one mind.

reubano commented 6 years ago

@vessemer great observation! I see that they provide evaluation code as well. So yes, adjusting it to fit our use case will be extremely useful!