search

NifTK / NiftyNet

[unmaintained] An open-source convolutional neural networks platform for research in medical image analysis and image-guided therapy
http://niftynet.io
Apache License 2.0
1.36k stars 404 forks source link

Loss stuck in DenseVnet training #350

Open KathrynSch opened 5 years ago

KathrynSch commented 5 years ago

Hi, I am trying to train the dense_vnet network on my own dataset (30 abdominal CT scans fro liver segmentation). The loss (dice_hinge.dice) got stuck at 0.24 after 1000 iteration and kept oscillating around that mean value for another 1000 iterations before I interrupted it. I then tried to overfit on a single sample (as a sanity check of the network). Same thing happened: stuck around 0.23 after 40 iterations (that I let go on for 600 iterations...). Other details: using resize sampler (to 144144144 volumes), no augmentation, adam optimizer as it is was set in the default config file (pretrained densevnet). Has this happened to anyone ? Any idea what might be going wrong here ? Thanks a lot for your help

ericspod commented 5 years ago

The discussion in #208 may be relevant to your case. It is hard to say without seeing your data set, but the issues of window size, batch size, initialization weights, or relative size of labels may be relevant to you.