mikevoets / jama16-retina-replication

JAMA 2016; 316(22) Replication Study
https://doi.org/10.1371/journal.pone.0217541
MIT License
110 stars 35 forks source link

Zeroed out confusion matrix and AUC = 0 during whole training #6

Closed temida closed 6 years ago

temida commented 6 years ago

Hi there, I am trying to reproduce your experiments in my university's lab and I'm observing some weird behaviour. After successfully following steps in readme regarding preprocessing (excluding Messidor for now) I run train.py without any parameters (as you describe it in the aforementioned readme). Cross entropy during training fluctuates naturally, but at the end of every epoch it turns out that Brier score, AUC and, what is more, confusion matrix - they are all filled with zeros :/ Due to the early stopping rule based on AUC training is then interrupted after epoch 10 (see attached log file). Any ideas what might have gone so wrong?

My environment:

^note: Python version is lower than specified in requirements, but I've noticed that the only Python feature forcing the scripts to require version 3.6 is f-string formatting, so I've modified all the print statements to use older {} formatting style.

ronnyshalev commented 6 years ago

@temida I was wondering if you could help me on the pre-processing step. After running the eyepacs.sh I was left with the folder structure as follows. It does not seem complete and I have no idea how to read the eyepacs.sh file (I have never used linux). If you could tell me what tree structure to expect, I will distribute the files manually.

data/ eyepacs/ pool/ train/ <----------- empty folder

            10_left.jpeg
            10_right.jpeg
            ... (35,127 image files)
mikevoets commented 6 years ago

@temida Do you still experience this problem with the latest release?

mikevoets commented 6 years ago

@ronnyshalev Please open another issue with your problem if you still have this problem with the latest code.