Vnet 3D segmentation: Train loss remains 0.99 all the time

[qianjiangcn]

Hi I am trying to train 3D segmentation network and using the Vnet config file in the github repo. But the result seems that it is not training at all since the loss remains all the time. Why would this happen and am I missing anything? Here is the config.ini I use. [T1] path_to_search = ../data filename_contains = T1 filename_not_contains = spatial_window_size = (32, 32, 32) pixdim = (1.0, 1.0, 1.0) axcodes=(A, R, S) interp_order = 3

[parcellation] path_to_search = ../data filename_contains = Mask filename_not_contains = spatial_window_size = (32, 32, 32) pixdim = (1.0, 1.0, 1.0) axcodes=(A, R, S) interp_order = 0

############################## system configuration sections [SYSTEM] cuda_devices = "" num_threads = 2 num_gpus = 1 model_dir = ./models/model_vnet

[NETWORK] name = vnet activation_function = prelu batch_size = 1 decay = 0 reg_type = L2

volume level preprocessing

volume_padding_size = 21

histogram normalisation

histogram_ref_file = ./example_volumes/monomodal_parcellation/standardisation_models.txt norm_type = percentile cutoff = (0.01, 0.99) normalisation = True whitening = True normalise_foreground_only=True foreground_type = otsu_plus multimod_foreground_type = and

queue_length = 1 window_sampling = uniform

[TRAINING] sample_per_volume = 32 rotation_angle = (-10.0, 10.0) scaling_percentage = (-10.0, 10.0) lr = 0.0001 loss_type = Dice starting_iter = 0 save_every_n = 5 max_iter = 10000 max_checkpoints = 20

[INFERENCE] border = (5, 5, 5)

inference_iter = 10

save_seg_dir = ./output/vnet output_interp_order = 0 spatial_window_size = (0, 0, 3)

############################ custom configuration sections [SEGMENTATION] image = T1 label = parcellation output_prob = False num_classes = 160 label_normalisation = True

The training looks like:

INFO:niftynet: training iter 1319, loss=1.0 (0.137058s) INFO:niftynet: training iter 1320, loss=0.9937499761581421 (0.134881s) INFO:niftynet: iter 1320 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt INFO:niftynet: training iter 1321, loss=0.9937499761581421 (0.146447s) INFO:niftynet: training iter 1322, loss=0.9941431879997253 (0.156982s) INFO:niftynet: training iter 1323, loss=0.9937499761581421 (0.149890s) INFO:niftynet: training iter 1324, loss=0.9937499761581421 (0.155439s) INFO:niftynet: training iter 1325, loss=0.9937499761581421 (0.148608s) INFO:niftynet: iter 1325 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt INFO:niftynet: training iter 1326, loss=0.993877649307251 (0.139796s) INFO:niftynet: training iter 1327, loss=0.9937499761581421 (0.133824s) INFO:niftynet: training iter 1328, loss=0.9941365122795105 (0.139889s) INFO:niftynet: training iter 1329, loss=0.9937499761581421 (0.142670s) INFO:niftynet: training iter 1330, loss=0.9939338564872742 (0.141735s) INFO:niftynet: iter 1330 saved: /ifshome/qjiang/Downloads/vnet/models/model_vnet/models/model.ckpt

Thank you very much!

[ericspod]

This may be related to #208. If your segmentations are really tiny then the background can dominate the dice calculation. Read through the discussion there to see if they have any relevant solutions.

[anmol4210]

Try changing the loss function from to GDSC