search

orobix / retina-unet

Retina blood vessel segmentation with a convolutional neural network
1.28k stars 468 forks source link

Is the 'random extracting' cause the different testing result? #7

Closed pythonokai closed 7 years ago

pythonokai commented 7 years ago

I try the same configuration file as 'test', but I can not achieve the same score 0.9790 as 'test'.

Why? I tried 2 times, First time get 0.9783, second time I get 0.9784. And I am trying the third time and it is not finish yet. But I guess this time will better than 0.9784, because that 'val_loss' is less than previous times(last improved: Epoch 00080: val_loss improved from 0.09076 to 0.09074)

So... Is the 'validation patches random extract from training datasets' cause the different result

Thank you for your work. And I have some question: First, I used GTX1060 6GB for this code, run as 'test' configuration will cost 900 sec per epoch, is that common? And how long you take with TITAN X? Second, is that 'loss' 'acc' 'val_loss' and 'val_acc' have some related? And some connection with AUC of ROC? (I will very apprecite if you teach me about that 4 data means.) Last, please let me know DL BBS or funds segmentation with DL BBS, and how I contact you?(Not this 'issue' way...) : )

Thank you very much and I am so sorry to asking quetions here no about the code.

Thank you.

dcorti commented 7 years ago

Hi,

Thank you for experimenting the code!

Yes, you're right. Since both the training and the validation datasets are obtained with random extraction of the patches, these datasets will be slightly different at every experiment. Therefore also the performance will be slightly different every time. I tried 5 different experiments reporting the following scores (rounded at 4th digit): 0.9788, 0.9792, 0.9787, 0.9790, 0.9791. Therefore, the results I show here are referred to the "average" run, the one reporting the score of 0.9790. I should try with more runs in order to get a minimum statistics. Otherwise I will fix the random seed, so the performance can be repeated more exactly. Please let me know if you get other results very different with respect to the numbers I got.

With my TITAN X I take roughly 10 minutes per epoch, so I guess 900 sec can be reasonable as well.

'loss' is the value of the loss function at the end of each epoch, while 'acc' is the (global) accuracy: the ratio between the number of pixels correctly classified and the total number of pixels. These are referred to the training dataset, while 'val_loss' and 'val_acc' are the same thing but related to the validation dataset. The AUC ROC is related to the accuracy, I think a nice explanation is reported here

I'm sorry, what's DL BBS?

pythonokai commented 7 years ago

Thank you for your rapid reply and your answers are very helpful to me.

I think I had a wrong expression, I mean forums about 'deep learning', best is about 'segmentation' (now we called forum...Not BBS...)