search

JohnleeHIT / Brats2019

[BrainLes2019] Multi-step cascaded network for brain tumor segmentations (tensorflow)
https://arxiv.org/abs/1908.05887
MIT License
95 stars 24 forks source link

after tranning 9000 epochs the matrice of dice is low(specially for ET) #1

Open Lightning980729 opened 4 years ago

Lightning980729 commented 4 years ago

Hello! All I changed is the "parameters.ini" , including phase \ traindata dir \ testdata dir . The traindata dir is just the unzip file of " MICCAI_BraTS_2019_Data_Training " Because I don't get the validation labels so i changed the testdatadir exacltly same as traindatadir.

The "parameters.ini" as follow

[param_setting]
; train / test / gen_map
phase = train
; batch size in training, 1
batch_size = 1
; batch size in training, 96
inputI_size = 96
; input channel number, 1
inputI_chn = 2
; output image size, 96
outputI_size = 96
; output channel, 8
output_chn = 2
; label rename map
rename_map = 0, 1, 2, 4
; volume resize ratio
resize_r = 1
; training data directory  traindata_dir =  /home/lixiangyu/Dataset/BraTS2019 /home/server/home/Dataset/BraTS2019
traindata_dir =  /data/yangjie/MICCAI_BraTS_2019_Data_Training
; checkpoint directory
chkpoint_dir = outcome/checkpoint
; learning rate of Adam, 1e-3 ，At first :0.001  when reach plateu decrease to 0.0005
learning_rate =0.001
; momentum term of Adam, 0.5
beta1 = 0.5
; training epoch, 10000
epoch =20000
; model name
model_name = ds_ft_hybrid_4ct.model
; model save interval
save_intval = 1000
; testing data  directory /home/lixiangyu/Dataset/mix/test /home/server/home/Dataset/mix/test
testdata_dir =  /data/yangjie/MICCAI_BraTS_2019_Data_Training
; labeling output directory
labeling_dir = outcome/label
; cube overlap factor： training:1 test:4
ovlp_ita =1

step=9000
Stages=6
Blocks=1
Columns=3
; Hard negative mining parameters
fg_ratio = 2
bg_ratio = 32

Then I just train the model , Run main.py in the command line . After tranning 9000 epochs , the matrice of dice is low(specially for ET, about 0.1) . the train.log file as follow

Configurations:
phase                          train
batch_size                     1
inputI_size                    96
inputI_chn                     2
outputI_size                   96
output_chn                     2
rename_map                     0, 1, 2, 4
resize_r                       1.0
traindata_dir                  /data/yangjie/MICCAI_BraTS_2019_Data_Training
chkpoint_dir                   outcome/checkpoint
learning_rate                  0.001
beta1                          0.5
epoch                          20000
model_name                     ds_ft_hybrid_4ct.model
save_intval                    1000
testdata_dir                   /data/yangjie/MICCAI_BraTS_2019_Data_Training
labeling_dir                   outcome/label
ovlp_ita                       1
step                           9000
Stages                         6
Blocks                         1
Columns                        3
fg_ratio                       2.0
bg_ratio                       32.0
focal_loss_flag                False
[WT, TC, ET]:  average dice: [0.441, 0.224, 0.021]  mean average dice : 0.22866666666666668 average sensitivity: [0.962, 0.951, 0.082]  mean average sensitivity : 0.6649999999999999
[WT, TC, ET]:  average dice: [0.375, 0.219, 0.034]  mean average dice : 0.20933333333333334 average sensitivity: [0.978, 0.944, 0.269]  mean average sensitivity : 0.7303333333333333
[WT, TC, ET]:  average dice: [0.5, 0.263, 0.025]  mean average dice : 0.26266666666666666 average sensitivity: [0.969, 0.972, 0.118]  mean average sensitivity : 0.6863333333333332
[WT, TC, ET]:  average dice: [0.434, 0.265, 0.052]  mean average dice : 0.25033333333333335 average sensitivity: [0.978, 0.966, 0.229]  mean average sensitivity : 0.7243333333333334
[WT, TC, ET]:  average dice: [0.479, 0.298, 0.04]  mean average dice : 0.2723333333333333 average sensitivity: [0.962, 0.937, 0.189]  mean average sensitivity : 0.6960000000000001
[WT, TC, ET]:  average dice: [0.556, 0.332, 0.046]  mean average dice : 0.3113333333333334 average sensitivity: [0.971, 0.951, 0.175]  mean average sensitivity : 0.699
[WT, TC, ET]:  average dice: [0.542, 0.282, 0.174]  mean average dice : 0.33266666666666667 average sensitivity: [0.964, 0.975, 0.258]  mean average sensitivity : 0.7323333333333334
[WT, TC, ET]:  average dice: [0.586, 0.328, 0.096]  mean average dice : 0.33666666666666667 average sensitivity: [0.98, 0.983, 0.322]  mean average sensitivity : 0.7616666666666667
[WT, TC, ET]:  average dice: [0.407, 0.245, 0.062]  mean average dice : 0.238 average sensitivity: [0.975, 0.974, 0.311]  mean average sensitivity : 0.7533333333333333
[WT, TC, ET]:  average dice: [0.516, 0.274, 0.07]  mean average dice : 0.2866666666666667 average sensitivity: [0.984, 0.989, 0.212]  mean average sensitivity : 0.7283333333333334
[WT, TC, ET]:  average dice: [0.535, 0.31, 0.065]  mean average dice : 0.3033333333333333 average sensitivity: [0.985, 0.968, 0.165]  mean average sensitivity : 0.706

the test.log file as follow

[WT, TC, ET]:  average dice: [0.513, 0.228, 0.041]  mean average dice : 0.26066666666666666 average sensitivity: [0.967, 0.979, 0.075]  mean average sensitivity : 0.6736666666666666
[WT, TC, ET]:  average dice: [0.445, 0.233, 0.087]  mean average dice : 0.255 average sensitivity: [0.983, 0.976, 0.211]  mean average sensitivity : 0.7233333333333333
[WT, TC, ET]:  average dice: [0.571, 0.274, 0.082]  mean average dice : 0.309 average sensitivity: [0.974, 0.985, 0.138]  mean average sensitivity : 0.699
[WT, TC, ET]:  average dice: [0.502, 0.285, 0.176]  mean average dice : 0.32099999999999995 average sensitivity: [0.981, 0.981, 0.267]  mean average sensitivity : 0.743
[WT, TC, ET]:  average dice: [0.57, 0.354, 0.165]  mean average dice : 0.363 average sensitivity: [0.96, 0.968, 0.253]  mean average sensitivity : 0.727
[WT, TC, ET]:  average dice: [0.626, 0.375, 0.137]  mean average dice : 0.3793333333333333 average sensitivity: [0.977, 0.976, 0.166]  mean average sensitivity : 0.7063333333333333
[WT, TC, ET]:  average dice: [0.629, 0.304, 0.292]  mean average dice : 0.4083333333333334 average sensitivity: [0.966, 0.988, 0.294]  mean average sensitivity : 0.7493333333333333
[WT, TC, ET]:  average dice: [0.656, 0.345, 0.274]  mean average dice : 0.425 average sensitivity: [0.984, 0.992, 0.358]  mean average sensitivity : 0.778
[WT, TC, ET]:  average dice: [0.49, 0.291, 0.198]  mean average dice : 0.3263333333333333 average sensitivity: [0.979, 0.983, 0.379]  mean average sensitivity : 0.7803333333333334
[WT, TC, ET]:  average dice: [0.593, 0.296, 0.19]  mean average dice : 0.35966666666666663 average sensitivity: [0.984, 0.995, 0.254]  mean average sensitivity : 0.7443333333333334
[WT, TC, ET]:  average dice: [0.604, 0.343, 0.177]  mean average dice : 0.3746666666666667 average sensitivity: [0.989, 0.988, 0.246]  mean average sensitivity : 0.741

Besides, when i am traning， the loss fluctuates. Do you konw the reason ? I would be appreciated if you could reply. Thanks a lot. Best wishes!

JohnleeHIT commented 4 years ago

Sorry for the delay, These days i am busy the a project, i still don't have the time to check the code, But i think that it might be the problem with the data, you can try training with only HGG data. Besides, the loss fluctuates is normal, because that loss is the loss of a single patch, not the whole volume.

Lightning980729 commented 4 years ago

Sorry for the delay, These days i am busy the a project, i still don't have the time to check the code, But i think that it might be the problem with the data, you can try training with only HGG data. Besides, the loss fluctuates is normal, because that loss is the loss of a single patch, not the whole volume.

Thanks for the reply. These day I try your advise to train with only HGG data. The result truely improved from the last one. Average dice increased by 10 points. But I cannot reappear your best result and there is still a big gap. After tranning 20000 epochs , the train.log file as follow

[WT, TC, ET]:  average dice: [0.592, 0.323, 0.082]  mean average dice : 0.3323333333333333 average sensitivity: [0.969, 0.973, 0.085]  mean average sensitivity : 0.6756666666666667
[WT, TC, ET]:  average dice: [0.422, 0.365, 0.17]  mean average dice : 0.319 average sensitivity: [0.982, 0.965, 0.247]  mean average sensitivity : 0.7313333333333333
[WT, TC, ET]:  average dice: [0.692, 0.433, 0.242]  mean average dice : 0.45566666666666666 average sensitivity: [0.969, 0.969, 0.246]  mean average sensitivity : 0.7280000000000001
[WT, TC, ET]:  average dice: [0.45, 0.344, 0.014]  mean average dice : 0.26933333333333337 average sensitivity: [0.991, 0.979, 0.022]  mean average sensitivity : 0.664
[WT, TC, ET]:  average dice: [0.682, 0.407, 0.341]  mean average dice : 0.4766666666666666 average sensitivity: [0.979, 0.984, 0.413]  mean average sensitivity : 0.7919999999999999
[WT, TC, ET]:  average dice: [0.598, 0.389, 0.283]  mean average dice : 0.42333333333333334 average sensitivity: [0.983, 0.984, 0.322]  mean average sensitivity : 0.763
[WT, TC, ET]:  average dice: [0.679, 0.438, 0.252]  mean average dice : 0.4563333333333333 average sensitivity: [0.983, 0.977, 0.254]  mean average sensitivity : 0.738
[WT, TC, ET]:  average dice: [0.678, 0.439, 0.262]  mean average dice : 0.45966666666666667 average sensitivity: [0.985, 0.982, 0.274]  mean average sensitivity : 0.747
[WT, TC, ET]:  average dice: [0.691, 0.52, 0.097]  mean average dice : 0.43599999999999994 average sensitivity: [0.98, 0.978, 0.066]  mean average sensitivity : 0.6746666666666666
[WT, TC, ET]:  average dice: [0.634, 0.314, 0.349]  mean average dice : 0.4323333333333333 average sensitivity: [0.993, 0.998, 0.47]  mean average sensitivity : 0.8203333333333335
[WT, TC, ET]:  average dice: [0.675, 0.473, 0.034]  mean average dice : 0.3940000000000001 average sensitivity: [0.987, 0.991, 0.022]  mean average sensitivity : 0.6666666666666666
[WT, TC, ET]:  average dice: [0.673, 0.499, 0.39]  mean average dice : 0.5206666666666667 average sensitivity: [0.974, 0.98, 0.406]  mean average sensitivity : 0.7866666666666666
[WT, TC, ET]:  average dice: [0.678, 0.423, 0.261]  mean average dice : 0.454 average sensitivity: [0.988, 0.994, 0.307]  mean average sensitivity : 0.763
[WT, TC, ET]:  average dice: [0.769, 0.513, 0.349]  mean average dice : 0.5436666666666666 average sensitivity: [0.983, 0.992, 0.346]  mean average sensitivity : 0.7736666666666667
[WT, TC, ET]:  average dice: [0.717, 0.501, 0.336]  mean average dice : 0.518 average sensitivity: [0.989, 0.99, 0.314]  mean average sensitivity : 0.7643333333333334
[WT, TC, ET]:  average dice: [0.787, 0.546, 0.446]  mean average dice : 0.5930000000000001 average sensitivity: [0.982, 0.99, 0.41]  mean average sensitivity : 0.794
[WT, TC, ET]:  average dice: [0.671, 0.572, 0.389]  mean average dice : 0.5439999999999999 average sensitivity: [0.982, 0.978, 0.364]  mean average sensitivity : 0.7746666666666666
[WT, TC, ET]:  average dice: [0.745, 0.573, 0.276]  mean average dice : 0.5313333333333333 average sensitivity: [0.982, 0.986, 0.223]  mean average sensitivity : 0.7303333333333333
[WT, TC, ET]:  average dice: [0.783, 0.598, 0.336]  mean average dice : 0.5723333333333334 average sensitivity: [0.983, 0.989, 0.277]  mean average sensitivity : 0.7496666666666667
[WT, TC, ET]:  average dice: [0.76, 0.642, 0.379]  mean average dice : 0.5936666666666667 average sensitivity: [0.985, 0.98, 0.33]  mean average sensitivity : 0.765

As you can see, the best result is [WT, TC, ET]: average dice: [0.787, 0.546, 0.446] Do you think it might be the problem with the parameters that you set on the "parameters.ini" ? Because of the limited computing resources, I didn't do more experiments.

zwlshine commented 4 years ago

@Lightning980729 Hello，train all data with 20000 epochs, how many times do you use? I train all data with 20000 epochs with learn rate 0.0001, patch size is 64, it caused more than 30 hours totally.

Lightning980729 commented 4 years ago

@Lightning980729 Hello，train all data with 20000 epochs, how many times do you use? I train all data with 20000 epochs with learn rate 0.0001, patch size is 64, it caused more than 30 hours totally.

It takes almost 5 days using one 1080ti, because I didn't split validation and i set the testing data the same as training data. Testing takes a lot time.

zwlshine commented 4 years ago

@Lightning980729 Hello，train all data with 20000 epochs, how many times do you use? I train all data with 20000 epochs with learn rate 0.0001, patch size is 64, it caused more than 30 hours totally.

It takes almost 5 days using one 1080ti, because I didn't split validation and i set the testing data the same as training data. Testing takes a lot time.

Reasonable time consumption. I split training dataset to 90% for train, 10% for test, and train 20000 epochs, patch size 96, on one 1080ti, the time is almost 48 hours.