questions about this code

[liuliu66]

Hi, thanks for your excellent work! I have several questions about how to use this code and reproduce your result step by step. Here is how I run this code on training by subset 0-8 and test on subset 9:

1. edit the config_training9.py code. indicate the training data path (subset 0-8), val data path (subset 9), test data path (also subset 9). Then I successfully got the npy files in each subsets.

2. run the run_training.sh, where I set test phase to test subset 9 using models from epoch 120-150 because I do not have enough gpus for test 150 times. During training, the loss seems fine. At this stage, I got 30 folders in DeepLung/detector/results/res18/retrft969/ which are named as 'val120' - 'val150'

3. run the evaluationScript/frocwrtdetpepchluna16.py, edit the eps as 'eps = range(120, maxeps+1, 1)'. Then I got lots of csv, txt and png files. But I found that the results are None. the froc curve is also nothing in it.

I do not know what is the problem. in running evaluationScript/frocwrtdetpepchluna16.py, an error occurs that no such file detector/results/res18/retrft969/val120/predanno-1.csv. I found the getcsv function writes csv file named predanno-13d.csv so I changed line 152 from '.csv' to '3d.csv'.

did I do something wrong? I used to do the 2D object detection rather than 3D so I am not familiar with this lung cancer data. Could you please help me? Thank you very much!

[wentaozhu]

If there is no modification from '.csv' to '3d.csv', the script will produce 'predanno-1.csv'.

Please check the folder detector/results/res18/retrft969/val120/. If there exists predanno-13d.csv, do not modify the file with '3d.csv'

[liuliu66]

@wentaozhu Hi, I still met a problem. when I test the model on subset9, the generated csv file in 'DeepLung/detector/results/res18/retrft969/val150/predanno-1.5d3.csv' seems fine, which is like this:

seriesuid	coordX	coordY	coordZ	probability
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-78.8897	-13.2915	-57.6934	0.999637
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-27.2572	33.74418	-170.349	0.992698
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-95.4039	-53.5542	-154.399	0.989167
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-58.8311	42.52674	-241.614	0.987959
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-135.921	21.49917	-237.37	0.986306
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-103.395	-46.6952	-173.838	0.962067
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-103.348	-6.73525	-151.02	0.883974
1.3.6.1.4.1.14519.5.2.1.6279.6001.558286136379689377915919180358	-56.091	45.09574	-239.567	0.862573

...

but when I run evaluationScript/frocwrtdetpepchluna16.py, the result is like this:

ep 150 detp -1.5 88 ep 150 detp -1 88 evaluationScript/annotations/annotations.csv evaluationScript/annotations/annotations.csv Total number of included nodule annotations: 1186 Total number of nodule annotations: 36378 Total number of included nodule annotations: 1186 Total number of nodule annotations: 36378 (1000, 10000) (0.07937846302095881, 16.0) [] /usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family [u'normal'] not found. Falling back to DejaVu Sans (prop.get_family(), self.defaultFamily[fontext])) (1000, 10000) (0.07937846302095881, 16.0) [] /usr/local/lib/python2.7/dist-packages/matplotlib/font_manager.py:1331: UserWarning: findfont: Font family [u'normal'] not found. Falling back to DejaVu Sans (prop.get_family(), self.defaultFamily[fontext])) [None, None] Traceback (most recent call last): File "evaluationScript/frocwrtdetpepchluna16.py", line 164, in getfroc(detp, eps) File "evaluationScript/frocwrtdetpepchluna16.py", line 161, in getfroc froclist[int((detpthresh-detp[0])/(detp[1]-detp[0]))] IndexError: list index out of range

I think it might be the froclist is empty so the result is None. could you please help check what is this problem? if I only wanna get froc and other evaluation metrics for subset9, how to do it?

Thanks!

[wentaozhu]

Maybe you can try to use the models here DeepLung /detector/dpnmodel instead of your trained model. If it runs well, the issue is from the training.

[liuliu66]

@liuliu66 I found the model in /detector/dpnmodel also works bad. But this model and my trained model could draw FROC curve fine. Then I run evaluationScript/frocwrtdetpepchluna16.py the noduleCADEvaluation() returns None. I think maybe there is a issue during evaluation. Because I only evaluate on subset9, I manually split the annotations.csv, annotations_excluded.csv and seriesuids.csv into subset9.

Did I do something wrong? Thanks!

[wentaozhu]

Did you run https://github.com/wentaozhu/DeepLung/blob/master/prepare.py? Yes, maybe it is cause by the splitting of annotation.csv.

[liuliu66]

@wentaozhu yes, I think I have run the prepare.py because there are lots of .npy files in LUNA16/subset9/, including _clean.npy, _extendbox.npy, _label.npy, _mask.npy, _origin.npy, _spacing.npy. Is it right? if something still wrong, I will debug the evaluation script step by step.

[liuliu66]

@wentaozhu Hi, thank you very much! I think I have run well. Sorry for bothering you again, I have some minor questions.

1. what is the meaning of detp = [-1.5, -1]. I guess it is the threshold for filtering detection results. But what the threshold for -1.5 and -1 in real world probability? How to transfer it?

2. In evaluationScript/noduleCADEvaluationLUNA16compare.py, what are the metrics meaning of return of evaluateCAD(). In other word, what the output of this evaluation script? like this:

0.6 0.7238095238095238 0.8 0.8476190476190476 0.8571428571428571 0.8857142857142857 0.9142857142857143 0.8040816326530613 Finished!

[wentaozhu]

It is the response of sigmoid activation function.

I think it is the FROC.

[penghuazhai]

@liuliu66 hello, I also meet the problem about donot have the file of predanno-1.csv . How da you slove the problem ? please give me some advice.

[wentaozhu]

@penghuazhai #91