search

MIC-DKFZ / nnDetection

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.
Apache License 2.0
543 stars 94 forks source link

[Question] Detection evaluation #122

Closed machur closed 1 year ago

machur commented 1 year ago

Hi @mibaumgartner,

I've run nnDetection on BRATS dataset to get familiar with the framework and I have the following questions:

  1. Do you save the learning curves somewhere? There are a lot of files and I'm not sure if I just can't find them or need to generate them by myself from training logs/pkls.
  2. I've been analyzing "analysis.csv" file and I don't understand the column "iou_tp" (3rd column). Would you please advise? I think all other metrics are pretty clear.
  3. From what I've seen the final results are available in file test_results\results_boxes.json, but these are aggregates. Do you save "partial" values per patient somewhere? I can't wrap my head around the transition from "analysis.csv" to "results_boxes.json". Any help would be much appreciated.
mibaumgartner commented 1 year ago

Dear @machur ,

(1) yes, there is a 'mlruns' folder inside the training directory or at path specified by MLFLOW_TRACKING_URI. Please refer to the mlflow documentation for information on how to visualize the logs. (2) The analysis is an oversimplification, e.g. the matching there does not include information on duplicate detections. As such the column is named "iou_tp" meaning that the IoU exceeds the IoU threshold but it might still be a duplicate prediction. (3) (AFAIK) All multi threshold detection metrics are aggregates there are no per patient results (the only exception are single threshold metrics which are not supported yet). That is why the analysis file was introduced for visual analysis of high scoring true positives, false positives etc. without going through all the cases manually. -> the Analysis file is not intended for evaluation! It uses a simplified matching algorithm and is based on a script I wrote some time back (i.e. it is not tested). Note: the pkl file contains slightly more info than the json, e.g. all the sensitivity values of the FROC curve

Best, Michael

mibaumgartner commented 1 year ago

Since there was no update for some time I’ll close this Issue for now. Please feel free to reopen this Issue if the problem persists or open a new one.