Question on detection performance evaluation with pre-trained models

[deepCAD]

I was trying to understand how the detection performance, i.e. FROC, was calculated. So I used pre-trained DPN models to generate detection results. For example, the subset0 results were generated with the following bash script:

CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --gpu '0,1,2,3' -b 16 --model dpn3d26 --resume fd0066.ckpt --test 1 --save-dir results/dpn3d26/retrft960/ --config config_training0

After obtaining detection results for all 10 folds, I used getcsv() in frocwrtdetpepchluna16.py to generate result csv files for each fold. Here I set detp = [-2] as suggested in the paper, and maxeps = 1. I obtained 39,576 candidates in total.

After concatenating 10 csv files, I used noduleCADEvaluationLUNA16.py to calculate FROC, here the annotation_filename is the annotations.csv downloaded from LUNA16 website seriesuids_filename includes 888 uids for all 10 subsets

I'm not sure how to set up annotations_excluded_filename, so just simply set it as the 3 cases in black_list item of config_training2.py

The FROC I obtained was around 0.74, which is much lower than the result reported in the paper. I guess I must have missed something, please take a look. I also attached CADAnalysis.txt for your reference. Thank you!

CADAnalysis.txt

[deepCAD]

I realized I should have used the annotations_excluded.csv provided by LUNA 16 website. this issue has been resolve.

[anruoxi123]

May I ask whether this FROC is calculated? I am a little confused, and I don't know which is FROC value. Thank you for your advice.

[wentaozhu]

You may check the DeepLung paper https://arxiv.org/abs/1801.09555