About the Config

[YilinLiu97]

Hi @josedolz, The training for my dataset works smoothly, but the results need some improvement. So I'm thinking about changing some hyper-parameters. But I am confused about some parts of the config file:

For "learning rate change Type", there seems no choices to choose from, same for the "Cost function values", "First Epoch Change LR", "Frequency Change LR".

TODO. To define some changes in the learning rate

learning Rate change Type = 0

Subvolumes (i.e. samples) sizes.

Validation equal to testing samples

sampleSize_Train = [27,27,27] sampleSize_Test = [45,45,45]

Cost function values

0:

1:

costFunction = 0 SoftMax temperature = 1.0

========= Learning rate ==========

L1 Regularization Constant = 1e-6 L2 Regularization Constant = 1e-4

TO check

The array size has to be equal to the total number of layers (i.e. CNNs + FCs + Classification layer)

Leraning Rate = [0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001 ]

Leraning Rate = [0.001]

First epoch to change learning rate

First Epoch Change LR = 1

Each how many epochs change learning rate

Frequency Change LR = 2

TODO. Add learning rate for each layer

Thanks!

[YilinLiu97]

And also for this one, I'm very confused about exactly which number to assign (1 or 2) in order to use the pre-trained weights. It seems that I should assign "2" but then the comment says "all layers will be initialized with pre-trained weights in case "weight_Initialization" is 1".

0, Classic

1: Delving (He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." ICCV'15)

2: Load Pre-trained

There is also the choice of which layers will be initialized with pre-trained weights. This is specified in #the variable "load weight layers". This can be either empty (i.e. all layers will be initialized with pre-#trained weights in case "weight_Initialization" is 1)

weight_Initialization_CNN = 1 weight_Initialization_FCN = 1

load weight layers = /trainedWeights # Next release

If using pre-trained models, specify the folder that contains the weights and the indexes of those weights to use

To ease the transfer between different softwares (i.e matlab for instance), and between different architectures,

the weights for each layer should be stored as a single file.

Right now weights have to be in .npy format

weights folderName = /Users/xxx/desktop/LiviaNET/trainedweights/

Same length as conv layers

weights trained indexes = [0,1,2]

weight_Initialization_Sec = 1

[josedolz]

Hi @YilinLiu97

To improve the performance one of the first things I would do is to change the network architecture given in the example, since it is actually very shallow (I gave that architecture just to run the example quickly).

For some of the parameters:

• "learning rate change Type" will define whether using some schedule to change the learning rate. So far no more options are allowed. The default way for changing the learning rate is defined by the two following parameters(which you can change as you want):

• "First Epoch Change LR", in which epoch you want to start to decrease your LR

• "Frequency Change LR", each how many epochs the LR must be decreased.

• For the weights initialization you are right. I did not update that part when I included the delving option.

There should be more features coming for the network, but since I have many things to do that may take long, sorry for that.

Best,

Jose.

[YilinLiu97]

Thanks for the reply!! Btw I have trouble loading the pre-trained weights. It seems to me that there are some tiny typos/bugs in LiviaNet.py, but even when I fixed those, I still got the following:

--- Weights initialization type: Transfer learning... Traceback (most recent call last): File "./networkTraining.py", line 82, in networkTraining(sys.argv[1:]) File "./networkTraining.py", line 72, in networkTraining networkModelName = generateNetwork(configIniName) File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/generateNetwork.py", line 128, in generateNetwork myParserConfigIni.tempSoftMax File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/LiviaNet.py", line 764, in createNetwork intermediate_ConnectedLayers)
File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/LiviaNet.py", line 256, in generateNetworkLayers dropoutRate File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/LiviaNet3DConvLayer.py", line 138, in init (convolvedOutput_Train, convolvedOutputShape_Train) = convolveWithKernel(self.W, filterShape, inputToConvTrain, inputToConvShapeTrain) File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/Modules/NeuralNetwork/layerOperations.py", line 63, in convolveWithKernel wReshapedForConv = W.dimshuffle(0,4,1,2,3) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/tensor/var.py", line 355, in dimshuffle pattern) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/tensor/elemwise.py", line 159, in init (i, j, len(input_broadcastable))) ValueError: new_order[1] is 4, but the input only has 1 axes.

[josedolz]

Can you please tell me what you modified in the code that according to you are bugs in the code?? I tried the code a lot of times with different configurations before deploying it here to be sure there were no errors(or the less possible). If there are some, it would be nice to be aware of them to correct them. Please include also your config file to check everything is ok. For example if you change the number of layers, or kernel size,you cannot use the weights I provide in the simple example and then when trying to create the net it will complain, as in this example.

Thanks.

[YilinLiu97]

Sure. 1) At first, I got this error,

File "/Users/xxx/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 764, in createNetwork intermediate_ConnectedLayers)
File "/Users/xxx/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 155, in generateNetworkLayers if len(weightsTrainedIdx) <> len(numberCNNLayers): NameError: global name 'weightsTrainedIdx' is not defined

What I modified: (in LiviaNet.py, 154:156)

if self.weight_Initialization_CNN == 2:   | if len(self.weightsTrainedIdx) <> len(numberCNNLayers):   | print(" ... WARNING!!!! Number of indexes specified for trained layers does not correspond with number of conv layers in the created architecture...")

2) But then I got this one:

self.weightsTrainedIdx is [0, 1, 2] len is 3 (These two lines are what I printed out for debugging) Traceback (most recent call last): File "./networkTraining.py", line 82, in networkTraining(sys.argv[1:]) File "./networkTraining.py", line 72, in networkTraining networkModelName = generateNetwork(configIniName) File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/generateNetwork.py", line 128, in generateNetwork myParserConfigIni.tempSoftMax File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/LiviaNet.py", line 766, in createNetwork intermediate_ConnectedLayers)
File "/Users/xxx/Downloads/LiviaNET-master/src/LiviaNet/LiviaNet.py", line 157, in generateNetworkLayers if len(self.weightsTrainedIdx) <> len(numberCNNLayers): TypeError: object of type 'int' has no len()

 I was confused about this error since self.weightsTrainedIdx should be a list instead of an int, just like what I printed out.

3) Lastly, I commented these two lines (154:156, in LiviaNet.py) and then I got the error described above.

[YilinLiu97]

I got these errors when I haven't modified the architectures.

############################################################################################################################################ ################################################# CREATION OF THE NETWORK ##################################################### ############################################################################################################################################

############## =================== General Options ================= ################ [General] networkName = liviaTest

Saving Options

folderName = LiviaNet_Test

############## =================== CNN_Architecture ================= ################ [CNN_Architecture] numkernelsperlayer = [10,20,30,100]

Kernels shapes: (Note, if kernel size is equal to 1 on one layer means that this layer is fully connected)

In this example there will be 3 conv layers and 1 fully connected layer (+ classification layer)

kernelshapes = [[3, 3, 3], [3, 3, 3], [3, 3, 3], [1]]

Intermediate layers to connect to the last conv layer (just before the first fully connected layer)

intermediateConnectedLayers = []

In the current implementation it does not support pooling (To be done...)

pooling_scales = [[1,1,1],[1,1,1],[1,1,1]]

Array size should be equal to number of fully connected (FC) layers + classification layer

dropout_Rates = [0.25,0.5]

Non-linear activations

Type: 0: Linear

1: ReLU

2: PReLU

3: LeakyReLU

activationType = 2

TODO. Include activation type for Softmax layer

Number of classes: background + classes to segment

n_classes = 9

------- Weights initialization -----------

There are some ways to initialize the weights. This is defined by the variable "weight_Initialization"

Here, there is a list of supported methods

0, Classic

1: Delving (He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." ICCV'15)

2: Load Pre-trained

----------

There is also the choice of which layers will be initialized with pre-trained weights. This is specified in the variable

"load weight layers". This can be either empty (i.e. all layers will be initialized with pre-trained weights in case

"weight_Initialization" is 1)

weight_Initialization_CNN = 2 weight_Initialization_FCN = 2

load weight layers = [] # Next release

If using pre-trained models, specify the folder that contains the weights and the indexes of those weights to use

To ease the transfer between different softwares (i.e matlab for instance), and between different architectures,

the weights for each layer should be stored as a single file.

Right now weights have to be in .npy format

weights folderName = /Users/xxx/Downloads/LiviaNET-master/trainedWeights

Same length as conv layers

weights trained indexes = [0,1,2]

weight_Initialization_Sec = 1

############## =================== Training Options ================= ################ [Training Parameters]

n_epochs=20

batch_size=5 number Of Epochs = 3 number Of SubEpochs = 2 number of samples at each SubEpoch Train = 1000

TODO. To define some changes in the learning rate

learning Rate change Type = 0

Subvolumes (i.e. samples) sizes.

Validation equal to testing samples

sampleSize_Train = [25,25,25] sampleSize_Test = [45,45,45]

Cost function values

0:

1:

costFunction = 0 SoftMax temperature = 1.0

========= Learning rate ==========

L1 Regularization Constant = 1e-6 L2 Regularization Constant = 1e-4

TO check

The array size has to be equal to the total number of layers (i.e. CNNs + FCs + Classification layer)

Leraning Rate = [0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001, 0.0001, 0.0001,0.0001, 0.0001 ]

Leraning Rate = [0.001]

First epoch to change learning rate

First Epoch Change LR = 1

Each how many epochs change learning rate

Frequency Change LR = 2

TODO. Add learning rate for each layer

========= Momentum ==========

Type of momentum

0: Classic

1: Nesterov

Momentum Type = 1 Momentum Value = 0.6

Use momentum normalized?

momentumNormalized = 1

======== Optimizer =====

Type: 0-> SGD

1-> RMSProp (TODO. Check why RMSProp complains....)

Optimizer Type = 1

In case we chose RMSProp

Rho RMSProp = 0.9 Epsilon RMSProp = 1e-4

Apply Batch normalization

0: False, 1: True

applyBatchNormalization = 1 BatchNormEpochs = 20

Apply padding to images

0: False, 1: True

applyPadding = 1

############################################################################################################################################ ################################################# TRAINING VALUES ##################################################### ############################################################################################################################################

[Training Images] imagesFolder = /Users/xxx/Downloads/LiviaNET-master/Dataset/MR/ GroundTruthFolder = /Users/xxx/Downloads/LiviaNET-master/Dataset/Label/

ROI folder will contain the ROI where to extract the pacthes and where to perform the segmentation.

Values of the ROI should be 0 (non interest) and 1 (region of interest)

ROIFolder = /Users/xxx/Downloads/LiviaNET-master/Dataset/ROI/

If you have no ROIs

ROIFolder = []

Type of images in the dataset

0: nifti format

1: matlab format

IMPORTANT: All the volumes should have been saved as 'vol'

imageTypes = 1

Indexes for training/validation images. Note that indexes should correspond to the position = inex + 1 in the folder,

since python starts indexing at 0

indexesForTraining = [0,1,2,3,4] indexesForValidation = [5]

[josedolz]

Ok, thanks for those comments. I'll check it asap and come you back.

Best,

[josedolz]

Hi @YilinLiu97

I was able to reproduce your error.

First of all, thank you for your comments, particularly for the ones of the small typos in the indexes of the weights. Actually it was during cleaning the code that I kept some parts I shouldn't. Here are the problems you found: 1 - You do not need to comment the lines 154-156 anymore. It is now updated to len(self.weightsTrainedIdx) <> numberCNNLayers. NumberCNNLayers must be an integer representing the number of CNNs defined in the network. Therefore having len(NumberCNNLayers) does not make sense. This has been updated in the code.

2- For the problem you get when loading the trained weights, it comes from the fact that I only provided weights for the convolutional layers, while you are trying to initialize also the weights of the fully connected layers with pre-trained weights, which do not exist. I just included the feature of using pre-trained weights in the convolutional layers. Just change in your config file weight_Initialization_FCN = 1. With this value equal to 2 it also crashed for me because of this. With this value to 1 it works (should work) like a charm now.

Best,

[YilinLiu97]

It works perfectly now!! Thanks so much!

When I elongated the network to 9 conv layers + 3 FC layers, as mentioned in your paper, I got 'nan' as the loss after a few rounds, even after I have lowered the learning rate a few order of magnitudes from 0.001. I'm wondering, have you encountered this and if so how did you solve it without using tricks like skip connection?

For this problem, my guess is that directly use your pre-trained weights (even just for the first 3 conv layers) and just train whatever layers left - freezing a few layers seems to be a common way for transfer learning. But as you said, the setup won't allow as we have to choose either 1 or 2 (weight initialization) for all conv layers. Is there any workarounds for this? Thanks!

[YilinLiu97]

Update: I was able to use the pre-trained weights for the first 3 conv layers and initialize the conv layers left with Delving by modifying a bit in the LiviaNet.py.

But got the following error:

** CREATING NETWORK ** --- Creating model (Reading parameters...) ** Starting creation model ** ------------------------ General ------------------------

• Network name: deep_LiviaNET
• Folder to save the outputs: DeepLiviaNET_Test ------------------------ CNN Architecture ------------------------
• Number of classes: 9
• Layers: [25, 25, 25, 50, 50, 50, 75, 75, 75, 100, 100, 100]
• Kernel sizes: [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [1], [1], [1]]
• Intermediate connected CNN layers: []
• Pooling: [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
• Dropout: [0.25, 0.5] --- Activation function: PReLU --- Weights initialization (CNN Layers): PreTrained --- Weights initialization (FCN Layers): Delving ------------------------ Training Parameters ------------------------
• Learning rate: [1e-07]
• Batch size: 5
• Apply batch normalization in 20 epochs ------------------------ Size of samples ------------------------
• Training: [27, 27, 27]
• Testing: [45, 45, 45] ...[STATUS]: Building the Network model...
• Shape of input subvolume (Training): (5, 1, 27, 27, 27)

• Shape of input subvolume (Testing): (5, 1, 45, 45, 45) ... WARNING!!!! Number of indexes specified for trained layers does not correspond with number of conv layers in the created architecture... --- [STATUS] --------- Creating layer 0 --------- --- Activation function: PReLU --- Weights initialization type: Transfer learning... --- [STATUS] --------- Creating layer 1 --------- --- Activation function: PReLU --- Weights initialization type: Transfer learning... --- [STATUS] --------- Creating layer 2 --------- --- Activation function: PReLU --- Weights initialization type: Transfer learning... --- [STATUS] --------- Creating layer 3 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: (5, 1, 27, 27, 27) ---> Output shape: [5, 50, 25, 25, 25] || kernel shape [50, 1, 3, 3, 3] ----- (Testing) Input shape: (5, 1, 45, 45, 45) ---> Output shape: [5, 50, 43, 43, 43] --- [STATUS] --------- Creating layer 4 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 25, 25, 25] ---> Output shape: [5, 50, 23, 23, 23] || kernel shape [50, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 43, 43, 43] ---> Output shape: [5, 50, 41, 41, 41] --- [STATUS] --------- Creating layer 5 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 23, 23, 23] ---> Output shape: [5, 50, 21, 21, 21] || kernel shape [50, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 41, 41, 41] ---> Output shape: [5, 50, 39, 39, 39] --- [STATUS] --------- Creating layer 6 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 21, 21, 21] ---> Output shape: [5, 75, 19, 19, 19] || kernel shape [75, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 39, 39, 39] ---> Output shape: [5, 75, 37, 37, 37] --- [STATUS] --------- Creating layer 7 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 19, 19, 19] ---> Output shape: [5, 75, 17, 17, 17] || kernel shape [75, 75, 3, 3, 3] ----- (Testing) Input shape: [5, 75, 37, 37, 37] ---> Output shape: [5, 75, 35, 35, 35] --- [STATUS] --------- Creating layer 8 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 17, 17, 17] ---> Output shape: [5, 75, 15, 15, 15] || kernel shape [75, 75, 3, 3, 3] ----- (Testing) Input shape: [5, 75, 35, 35, 35] ---> Output shape: [5, 75, 33, 33, 33] --- Starting to create the fully connected layers.... --- [STATUS] --------- Creating layer 9 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 15, 15, 15] ---> Output shape: [5, 100, 15, 15, 15] || kernel shape [100, 75, 1, 1, 1] ----- (Testing) Input shape: [5, 75, 33, 33, 33] ---> Output shape: [5, 100, 33, 33, 33] --- [STATUS] --------- Creating layer 10 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 15, 15, 15] ---> Output shape: [5, 100, 15, 15, 15] || kernel shape [100, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 33, 33, 33] ---> Output shape: [5, 100, 33, 33, 33] --- [STATUS] --------- Creating layer 11 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 15, 15, 15] ---> Output shape: [5, 100, 15, 15, 15] || kernel shape [100, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 33, 33, 33] ---> Output shape: [5, 100, 33, 33, 33] ----- (Classification layer) kernel shape [9, 100, 1, 1, 1] --- [STATUS] --------- Creating layer 11 --------- --- Activation function: Linear --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 15, 15, 15] ---> Output shape: [5, 9, 15, 15, 15] || kernel shape [9, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 33, 33, 33] ---> Output shape: [5, 9, 33, 33, 33] ------- Initializing network training parameters........... --- Optimizer: Stochastic gradient descent (SGD) ----------------- Starting compilation process ----------------- --- Cost function: negativeLogLikelihood --- Optimizer: Stochastic gradient descent (SGD) Traceback (most recent call last): File "./networkTraining.py", line 82, in networkTraining(sys.argv[1:]) File "./networkTraining.py", line 72, in networkTraining networkModelName = generateNetwork(configIniName) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/generateNetwork.py", line 147, in generateNetwork myLiviaNet3D.compileTheanoFunctions() File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 645, in compileTheanoFunctions updates = self.getUpdatesOfTrainableParameters(cost, paramsTraining, numberParamsPerLayer) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 414, in getUpdatesOfTrainableParameters updates = optionsOptimizer[self.optimizerType]() File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 403, in SGD updates = self.updateParams_SGD(cost, paramsTraining, numberParamsPerLayer) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 423, in updateParams_SGD grads = T.grad(cost, paramsTraining) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/gradient.py", line 539, in grad handle_disconnected(elem) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/theano/gradient.py", line 526, in handle_disconnected raise DisconnectedInputError(message) theano.gradient.DisconnectedInputError:
  Backtrace when that variable is created:

  File "./networkTraining.py", line 82, in networkTraining(sys.argv[1:]) File "./networkTraining.py", line 72, in networkTraining networkModelName = generateNetwork(configIniName) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/generateNetwork.py", line 128, in generateNetwork myParserConfigIni.tempSoftMax File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 800, in createNetwork intermediate_ConnectedLayers) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet.py", line 189, in generateNetworkLayers dropoutRate File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet3DConvLayer.py", line 120, in init rng File "/Users/x/Desktop/LiviaNET/src/LiviaNet/LiviaNet3DConvLayer.py", line 216, in passInputThroughLayerElements inputSampleShape_Train) File "/Users/x/Desktop/LiviaNET/src/LiviaNet/Modules/NeuralNetwork/layerOperations.py", line 97, in applyBn gBn = theano.shared(value=gBn_values, borrow=True)

[josedolz]

Hi @YilinLiu97

If you want to use pre-trained weights for the 9 conv layers, I suggest you to use that architecture with the demo data, save those weights and use them as pre-trained ones for your data. However, I am sure the NaN problem comes from your data and this will not solve it. I would inspect the content of both the MRI and GT samples that are sent to the trainer, since I guess there should be something wrong there.

Further, I am sorry, but if you modify the code I cannot guarantee it will work. Unless you are very comfortable with Theano, I wouldn't recommend to change things, because it is difficult to debug. This error seems to me that can come from the fact that you missed to connect some of the conv layers (either the weights or bias, or maybe both).

Best

[YilinLiu97]

Hi @josedolz , thanks for the reply! Actually I used that architecture with the demo data and still got the nan error. I also observed that the option of resuming training from epochX seems not working for me - it always gives me nan.

** CREATING NETWORK ** --- Creating model (Reading parameters...) ** Starting creation model ** ------------------------ General ------------------------

• Network name: deep_LiviaNET
• Folder to save the outputs: DeepLiviaNET_MICCAI_Test ------------------------ CNN Architecture ------------------------
• Number of classes: 9
• Layers: [25, 25, 25, 50, 50, 50, 75, 75, 75, 100, 100, 100]
• Kernel sizes: [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [1], [1], [1]]
• Intermediate connected CNN layers: []
• Pooling: [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
• Dropout: [0.25, 0.5] --- Activation function: PReLU --- Weights initialization (CNN Layers): Delving --- Weights initialization (FCN Layers): Delving ------------------------ Training Parameters ------------------------
• Learning rate: [0.001]
• Batch size: 5
• Apply batch normalization in 20 epochs ------------------------ Size of samples ------------------------
• Training: [25, 25, 25]
• Testing: [45, 45, 45] ...[STATUS]: Building the Network model...
• Shape of input subvolume (Training): (5, 1, 25, 25, 25)

• Shape of input subvolume (Testing): (5, 1, 45, 45, 45) --- [STATUS] --------- Creating layer 0 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: (5, 1, 25, 25, 25) ---> Output shape: [5, 25, 23, 23, 23] || kernel shape [25, 1, 3, 3, 3] ----- (Testing) Input shape: (5, 1, 45, 45, 45) ---> Output shape: [5, 25, 43, 43, 43] --- [STATUS] --------- Creating layer 1 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 25, 23, 23, 23] ---> Output shape: [5, 25, 21, 21, 21] || kernel shape [25, 25, 3, 3, 3] ----- (Testing) Input shape: [5, 25, 43, 43, 43] ---> Output shape: [5, 25, 41, 41, 41] --- [STATUS] --------- Creating layer 2 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 25, 21, 21, 21] ---> Output shape: [5, 25, 19, 19, 19] || kernel shape [25, 25, 3, 3, 3] ----- (Testing) Input shape: [5, 25, 41, 41, 41] ---> Output shape: [5, 25, 39, 39, 39] --- [STATUS] --------- Creating layer 3 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 25, 19, 19, 19] ---> Output shape: [5, 50, 17, 17, 17] || kernel shape [50, 25, 3, 3, 3] ----- (Testing) Input shape: [5, 25, 39, 39, 39] ---> Output shape: [5, 50, 37, 37, 37] --- [STATUS] --------- Creating layer 4 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 17, 17, 17] ---> Output shape: [5, 50, 15, 15, 15] || kernel shape [50, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 37, 37, 37] ---> Output shape: [5, 50, 35, 35, 35] --- [STATUS] --------- Creating layer 5 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 15, 15, 15] ---> Output shape: [5, 50, 13, 13, 13] || kernel shape [50, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 35, 35, 35] ---> Output shape: [5, 50, 33, 33, 33] --- [STATUS] --------- Creating layer 6 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 50, 13, 13, 13] ---> Output shape: [5, 75, 11, 11, 11] || kernel shape [75, 50, 3, 3, 3] ----- (Testing) Input shape: [5, 50, 33, 33, 33] ---> Output shape: [5, 75, 31, 31, 31] --- [STATUS] --------- Creating layer 7 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 11, 11, 11] ---> Output shape: [5, 75, 9, 9, 9] || kernel shape [75, 75, 3, 3, 3] ----- (Testing) Input shape: [5, 75, 31, 31, 31] ---> Output shape: [5, 75, 29, 29, 29] --- [STATUS] --------- Creating layer 8 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 9, 9, 9] ---> Output shape: [5, 75, 7, 7, 7] || kernel shape [75, 75, 3, 3, 3] ----- (Testing) Input shape: [5, 75, 29, 29, 29] ---> Output shape: [5, 75, 27, 27, 27] --- Starting to create the fully connected layers.... --- [STATUS] --------- Creating layer 9 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 75, 7, 7, 7] ---> Output shape: [5, 100, 7, 7, 7] || kernel shape [100, 75, 1, 1, 1] ----- (Testing) Input shape: [5, 75, 27, 27, 27] ---> Output shape: [5, 100, 27, 27, 27] --- [STATUS] --------- Creating layer 10 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 7, 7, 7] ---> Output shape: [5, 100, 7, 7, 7] || kernel shape [100, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 27, 27, 27] ---> Output shape: [5, 100, 27, 27, 27] --- [STATUS] --------- Creating layer 11 --------- --- Activation function: PReLU --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 7, 7, 7] ---> Output shape: [5, 100, 7, 7, 7] || kernel shape [100, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 27, 27, 27] ---> Output shape: [5, 100, 27, 27, 27] ----- (Classification layer) kernel shape [9, 100, 1, 1, 1] --- [STATUS] --------- Creating layer 11 --------- --- Activation function: Linear --- Weights initialization type: Delving ----- (Training) Input shape: [5, 100, 7, 7, 7] ---> Output shape: [5, 9, 7, 7, 7] || kernel shape [9, 100, 1, 1, 1] ----- (Testing) Input shape: [5, 100, 27, 27, 27] ---> Output shape: [5, 9, 27, 27, 27] ------- Initializing network training parameters........... --- Optimizer: Stochastic gradient descent (SGD) ----------------- Starting compilation process ----------------- --- Cost function: negativeLogLikelihood --- Optimizer: Stochastic gradient descent (SGD) ...Compiling the training function... ...The training function was compiled... ...Compiling the testing function... ...The testing function was compiled... ..Folder Networks created... ..Folder to store predictions (Validation) created... ..Folder to store predictions (Testing) created... Network model saved in /Users/xxx/Downloads/LiviaNET_Test/src/outputFiles/DeepLiviaNET_MICCAI_Test/Networks as deep_LiviaNET_Epoch0 ** NETWORK CREATED ** ** STARTING NETWORK TRAINING ** **** STARTING TRAINING ** ** Starting training model (Reading parameters) ** --- Training model (Reading parameters...) --- Do training in 3 epochs with 2 subEpochs each... -------- Reading Images names used in training/validation ------------- ================== Images for training ================ Image(0): MR_Img1.mat | GT: GT_Img1.mat | ROI ROI_Img1.mat Image(1): MR_Img2.mat | GT: GT_Img2.mat | ROI ROI_Img2.mat Image(2): MR_Img3.mat | GT: GT_Img3.mat | ROI ROI_Img3.mat Image(3): MR_Img4.mat | GT: GT_Img4.mat | ROI ROI_Img4.mat Image(4): MR_Img5.mat | GT: GT_Img5.mat | ROI ROI_Img5.mat ================== Images for validation ================ Image(0): MR_Img6.mat | GT: GT_Img6.mat | ROI ROI_Img6.mat

  ... Loading model from /Users/xxx/Downloads/LiviaNET_Test/src/outputFiles/DeepLiviaNET_MICCAI_Test/Networks/deep_LiviaNET_Epoch0 ... Network architecture successfully loaded.... ============== EPOCH: 1/3 ================= --- SubEPOCH: 1/2 ... Get samples for subEpoch... ... getting 200 samples per subject... ...Processing subject: 1. 20.0 % of the whole training set... ...Processing subject: 2. 40.0 % of the whole training set... ...Processing subject: 3. 60.0 % of the whole training set... ...Processing subject: 4. 80.0 % of the whole training set... ...Processing subject: 5. 100.0 % of the whole training set... ---------- Cost of this subEpoch: nan Thanks!

[YilinLiu97]

I forgot to mention that for this one, I didn't modify anything in the source codes (For testing, I re-downloaded your codes). I really hope to get the "deeper" network working...

Here is the config file:

############################################################################################################################################ ################################################# CREATION OF THE NETWORK ##################################################### ############################################################################################################################################

############## =================== General Options ================= ################ [General] networkName = deep_LiviaNET

Saving Options

folderName = DeepLiviaNET_MICCAI_Test

############## =================== CNN_Architecture ================= ################ [CNN_Architecture] numkernelsperlayer = [25,25,25,50,50,50,75,75,75,100,100,100]

Kernels shapes: (Note, if kernel size is equal to 1 on one layer means that this layer is fully connected)

In this example there will be 3 conv layers and 1 fully connected layer (+ classification layer)

kernelshapes = [[3, 3, 3],[3,3,3],[3,3,3],[3,3,3],[3,3,3],[3,3,3],[3,3,3], [3, 3, 3], [3, 3, 3], [1],[1],[1]]

Intermediate layers to connect to the last conv layer (just before the first fully connected layer)

intermediateConnectedLayers = []

In the current implementation it does not support pooling (To be done...)

pooling_scales = [[1,1,1],[1,1,1],[1,1,1]]

Array size should be equal to number of fully connected (FC) layers + classification layer

dropout_Rates = [0.25,0.5]

Non-linear activations

Type: 0: Linear

1: ReLU

2: PReLU

3: LeakyReLU

activationType = 2

TODO. Include activation type for Softmax layer

Number of classes: background + classes to segment

n_classes = 9

------- Weights initialization -----------

There are some ways to initialize the weights. This is defined by the variable "weight_Initialization"

Here, there is a list of supported methods

0, Classic

1: Delving (He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." ICCV'15)

2: Load Pre-trained

----------

There is also the choice of which layers will be initialized with pre-trained weights. This is specified in the variable

"load weight layers". This can be either empty (i.e. all layers will be initialized with pre-trained weights in case

"weight_Initialization" is 1)

weight_Initialization_CNN = 1 weight_Initialization_FCN = 1

load weight layers = /trainedWeights # Next release

If using pre-trained models, specify the folder that contains the weights and the indexes of those weights to use

To ease the transfer between different softwares (i.e matlab for instance), and between different architectures,

the weights for each layer should be stored as a single file.

Right now weights have to be in .npy format

weights folderName = /Users/xxx/desktop/LiviaNET/trainedweights/

Same length as conv layers

weights trained indexes = [0,1,2]

weight_Initialization_Sec = 1

############## =================== Training Options ================= ################ [Training Parameters]

n_epochs=20

batch_size=5 number Of Epochs = 3 number Of SubEpochs = 2 number of samples at each SubEpoch Train = 1000

TODO. To define some changes in the learning rate

learning Rate change Type = 0

Subvolumes (i.e. samples) sizes.

Validation equal to testing samples

sampleSize_Train = [25,25,25] sampleSize_Test = [45,45,45]

Cost function values

0:

1:

costFunction = 0 SoftMax temperature = 1.0

========= Learning rate ==========

L1 Regularization Constant = 1e-6 L2 Regularization Constant = 1e-4

TO check

The array size has to be equal to the total number of layers (i.e. CNNs + FCs + Classification layer)

Leraning Rate = [0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.0000001,0.$

Leraning Rate = [0.00001]

First epoch to change learning rate

First Epoch Change LR = 1

Each how many epochs change learning rate

Frequency Change LR = 2

TODO. Add learning rate for each layer

========= Momentum ==========

Type of momentum

0: Classic

1: Nesterov

Momentum Type = 1 Momentum Value = 0.6

Use momentum normalized?

momentumNormalized = 1

======== Optimizer =====

Type: 0-> SGD

1-> RMSProp (TODO. Check why RMSProp complains....)

Optimizer Type = 0

In case we chose RMSProp

Rho RMSProp = 0.9 Epsilon RMSProp = 1e-4

Apply Batch normalization

0: False, 1: True

applyBatchNormalization = 1 BatchNormEpochs = 20

Apply padding to images

0: False, 1: True

applyPadding = 1

############################################################################################################################################ ################################################# TRAINING VALUES ##################################################### ############################################################################################################################################

[Training Images] imagesFolder = /Users/xxx/desktop/LiviaNET/Dataset/MR/ GroundTruthFolder = /Users/xxx/desktop/LiviaNET/Dataset/Label/

ROI folder will contain the ROI where to extract the pacthes and where to perform the segmentation.

Values of the ROI should be 0 (non interest) and 1 (region of interest)

ROIFolder = /Users/xxx/desktop/LiviaNET/Dataset/ROI/

If you have no ROIs

ROIFolder = []

Type of images in the dataset

0: nifti format

1: matlab format

IMPORTANT: All the volumes should have been saved as 'vol'

imageTypes = 1

Indexes for training/validation images. Note that indexes should correspond to the position = inex + 1 in the folder,

since python starts indexing at 0

indexesForTraining = [0,1,2,3,4] indexesForValidation = [5]

[YilinLiu97]

Some updates...

============== EPOCH: 1/3 ================= --- SubEPOCH: 1/2 ... Get samples for subEpoch... ... getting 200 samples per subject... ...Processing subject: 1. 20.0 % of the whole training set... ...Processing subject: 2. 40.0 % of the whole training set... ...Processing subject: 3. 60.0 % of the whole training set... ...Processing subject: 4. 80.0 % of the whole training set... ...Processing subject: 5. 100.0 % of the whole training set... ---------- Cost of this subEpoch: 2.76706286669 --- SubEPOCH: 2/2 ... Get samples for subEpoch... ... getting 200 samples per subject... ...Processing subject: 1. 20.0 % of the whole training set... ...Processing subject: 2. 40.0 % of the whole training set... ...Processing subject: 3. 60.0 % of the whole training set... ...Processing subject: 4. 80.0 % of the whole training set... ...Processing subject: 5. 100.0 % of the whole training set... ---------- Cost of this subEpoch: 2.76206525445 ---------- Training on Epoch #0 finished ---------- ---------- Cost of Epoch: 2.76456406057 / Mean training error 2.76456406057

---

** Starting validation ** ------------- Segmenting subject: MR_Img6.mat ....total: 1/1... ------------- ... Saving segmentation result... ... Image succesfully saved... ... Saving prob map for class 1... ... Image succesfully saved... ... Saving prob map for class 2... ... Image succesfully saved... ... Saving prob map for class 3... ... Image succesfully saved... ... Saving prob map for class 4... ... Image succesfully saved... ... Saving prob map for class 5... ... Image succesfully saved... ... Saving prob map for class 6... ... Image succesfully saved... ... Saving prob map for class 7... ... Image succesfully saved... ... Saving prob map for class 8... ... Image succesfully saved... ... Computing Dice scores: -------------- DSC (Class 1) : 0.0100566487912 -------------- DSC (Class 2) : 0.00435581907617 -------------- DSC (Class 3) : 0.00441681217125 -------------- DSC (Class 4) : 0.00016433422003 -------------- DSC (Class 5) : 0.00606102799949 -------------- DSC (Class 6) : 0.00599444724886 -------------- DSC (Class 7) : 0.00960786097716 -------------- DSC (Class 8) : 0.00318045358618 ** Validation DONE ** Network model saved in /Users/xxx/Downloads/LiviaNET_Test/src/outputFiles/DeepLiviaNET_MICCAI_Test/Networks as deep_LiviaNET_Epoch1 ============== EPOCH: 2/3 ================= --- SubEPOCH: 1/2 ... Get samples for subEpoch... ... getting 200 samples per subject... ...Processing subject: 1. 20.0 % of the whole training set... ...Processing subject: 2. 40.0 % of the whole training set... ...Processing subject: 3. 60.0 % of the whole training set... ...Processing subject: 4. 80.0 % of the whole training set... ...Processing subject: 5. 100.0 % of the whole training set... ---------- Cost of this subEpoch: 2.75880724072 --- SubEPOCH: 2/2 ... Get samples for subEpoch... ... getting 200 samples per subject... ...Processing subject: 1. 20.0 % of the whole training set... ...Processing subject: 2. 40.0 % of the whole training set... ...Processing subject: 3. 60.0 % of the whole training set... ...Processing subject: 4. 80.0 % of the whole training set... ...Processing subject: 5. 100.0 % of the whole training set... ---------- Cost of this subEpoch: nan ---------- Training on Epoch #1 finished ---------- ---------- Cost of Epoch: nan / Mean training error nan

---

** Starting validation ** ------------- Segmenting subject: MR_Img6.mat ....total: 1/1... -------------

[mitrasafari]

hi I have the same problem as this,could you help me with the config file? thanks

[josedolz]

Hi @mitrasafari

Please have a look to #10

I added the config file employed in the paper, with which you should be able to reproduce the results, as in #10 .

Let me know whether that solves your problem.

Jose

[mitrasafari]

Hi @josedolz HI Thanks for your reply I use your config file but it just create until layer 11 and after that say (Type Error:<TensorType(float32, scalar)>…) I attach screenshot of my cmd that you can see the exact error I use " THEANO_FLAGS='floatX=float32' python ./networkTraining.py ./LiviaNET_Config.ini 0" for configuration but it say THEANO_FLAGS is not recognize. actually i am not expert in this area would you please explain in more details Thanks so much

[josedolz]

Hi @mitrasafari

Can you show me the terminal where it complains about THEANO_FLAGS? This is strange.

Anyway, the problem comes from some incompatibilities between some tensor types and numpy values/arrays. If the Theano configuration is set properly, this will not cause the code to crash. However, it seems this is not your case. I have to work on this, but unfortunately I have not time so far.

Some work-around: -1) With the THEANO_FLAGS the problem should be gone. -2) Run the code in GPU. I have seen this problem only when running on CPU. Further, making the whole training on CPU will take more than one week. -3) Set floatX=float32 in the theanorc file.

Hope this helps.

Best

[mitrasafari]

Hi@josedolz sorry to reply late I attach my cmd when i use "THEANO_FLAGS='floatX=float32' python ./networkTraining.py ./LiviaNET_Config.ini 0" I'm pretty sure the problem is my THEANO and i try different type of installation, but still it is not working

[josedolz]

Hi @mitrasafari

This is a problem with your environment, not with the code.

I have found people reporting similar problems here:

https://stackoverflow.com/questions/40157515/command-to-run-theano-on-gpu-windows https://stackoverflow.com/questions/41436068/theano-flags-command-not-found

Basically some of these options tell you to run this instead: set THEANO_FLAGS="floatX=float32" & python xxxxxxx.py