Dice normalization?

[sagarhukkire]

Hi @faustomilletari

I trained Vnet on my data and applied test to get automatic labels. I got same.But the automatic labels are not full available seems they are cut in portion ( you can see in picture)

But I am stuck with few points ,hope you will guide me.

1. I kept batch size as 1 and during forward pass in pylayer I made change for dice normalization

   top[0].data[0]=np.sum(dice)/float(bottom[0].data.shape[0])

Am I missing something ? Thanks in advance for all your help

[faustomilletari]

there might be something connected with connected components analysis. if you are using the original data manager code comment this portion https://github.com/faustomilletari/Hough-CNN/blob/master/DataManager.py#L187-L199 https://github.com/faustomilletari/Hough-CNN/blob/master/DataManager.py#L187-L199 to have no connected component analysis.

It still looks weird though. therefore I would suggest you to additionally comment also https://github.com/faustomilletari/Hough-CNN/blob/master/DataManager.py#L187-L201 https://github.com/faustomilletari/Hough-CNN/blob/master/DataManager.py#L187-L201 such that you will see the probability map of your foreground.

Regards, Fausto

On Mar 13, 2017, at 4:59 PM, sagarax009 notifications@github.com wrote:

Hi @faustomilletari https://github.com/faustomilletari I trained Vnet on my data and applied test to get automatic labels. I got same.But the automatic labels are not full available seems they are cut in portion ( you can see in picture)

But I am stuck with few points ,hope you will guide me.

I kept batch size as 1 and during forward pass in pylayer I made change for dice normalization top[0].data[0]=np.sum(dice)/float(bottom[0].data.shape[0])

Am I missing something ? Thanks in advance for all your help

https://cloud.githubusercontent.com/assets/20017611/23874990/1528e180-0838-11e7-8a5d-1b4d20efa8fb.png — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvmxXu2g5y7znm2DgF0yeVuTAJN4Cks5rla4rgaJpZM4MbzzB.

[sagarhukkire]

Thanks @faustomilletari

I will try it and let you know how it goes .

[faustomilletari]

I linked by mistake the code from a different repo. but the content is similar.

On Mar 13, 2017, at 5:09 PM, sagarax009 notifications@github.com wrote:

Thanks @faustomilletari https://github.com/faustomilletari I will try it and let it know how it goes .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19#issuecomment-286244681, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvo7sDLOyJ9iy-6pbhdua8J3hJJJbks5rlbCigaJpZM4MbzzB.

[sagarhukkire]

hi @faustomilletari

I commented Connect component code and re ran but it seems no improvement in results.(PFA)

I have small doubt during forward I am making top[0].data[0]=np.sum(dice)/float(bottom[0].data.shape[0])

do I need to do any changes in backward or is it okay ? Is this reason to this problem?

[gattia]

Im assuming this is a 3D image, what are the dimensions?

I ask because Vnet has image augmentation, yes? If rotations are occurring about the y or x axis of this image instead of the z-axis and there are a really small number of z-slices I wonder if something funky is going on during training and it's learning to only seeing part of the segmentation depending on the slice.

Do all of the slices of segmentation look the same?

It might be informative to see some of the unlabeled data.

[sagarhukkire]

hi @faustomilletari

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

Let me put all steps which I followed ,so you can guide me if I am wrong somewhere

1. I have data and labels in .nrrd format ,since sitk read same 2 . Batch size is 2 and dice loss is normalized
2. CC code is commented

Thanks in advance

[faustomilletari]

Hello,

I have see that that you are experiencing before. One thing that you can do, just to test, is to take advantage of the fully convolutional nature of this network and change the prototxt ONLY FOR TESTING to accept inputs which are bigger in x and you direction (if you have memory problems (you shouldn't) you can lower the size in z) (the weight of the network will be still loaded without problems!) and then feed in the network a PADDED image with quite a bit of zero padding all around in x and you direction.

If then you see more segmentation appearing I think we have to start to consider some aspects more in depth, like padding throughout the network, which is something controversial... (So far I could not find problems because of that)

Fausto Milletarì Sent from my iPhone

On 14 Mar 2017, at 10:55, sagarax009 notifications@github.com wrote:

hi @faustomilletari

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

Let me put all steps which I followed ,so you can guide me if I am wrong somewhere

I have data and labels in .nrrd format ,since sitk read same 2 . Batch size is 2 and dice loss is normalized CC code is commented Thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[faustomilletari]

My phone decided to correct y in you. Or course I meant y direction not you direction. Please excuse other typos.

Fausto Milletarì Sent from my iPhone

On 14 Mar 2017, at 10:55, sagarax009 notifications@github.com wrote:

hi @faustomilletari

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

Let me put all steps which I followed ,so you can guide me if I am wrong somewhere

I have data and labels in .nrrd format ,since sitk read same 2 . Batch size is 2 and dice loss is normalized CC code is commented Thanks in advance

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sagarhukkire]

@faustomilletari

If i understood correctly , currently I have trained model ,but the issue is appearing when I pass the training images.

1) Instead I pass test images to network ,by padding them with zeros in x and y direction. is that right?

[faustomilletari]

Yes,

Pass bigger images (you need to change input size in prototxt).

Let's see what happens...

Also visualizing probabilities for foreground will help to understand what's wrong.

Fausto Milletarì Sent from my iPhone

On 14 Mar 2017, at 11:30, sagarax009 notifications@github.com wrote:

@faustomilletari

If i understood correctly , currently I have trained model ,but the issue is appearing when I pass the training images.

Instead I pass test images to network ,by padding them with zeros in x and y direction. is that right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sagarhukkire]

hi @faustomilletari

I made changes in test.prototxt file with dimension 256,256,128

but I am getting below error

File "/net/linse8-sn/no_backup_01/d1202/VNet-master/VNet.py", line 195, in test net.blobs['data'].data[...] = btch ValueError: could not broadcast input array from shape (1,1,128,128,64) into shape (1,1,256,256,128)

Do I need to train model again by changing train.prototxt file too or am I missing something?

thanks

[faustomilletari]

No, I don’t think so. I think you might need to be sure that:

1) you are using the correct prototxt for testing -- “test” prototxt 2) you are feeding into the network data that is padded. the padding has to be done manually ( :( ) i think you can use numpy to do it without issues.

check with btch.shape if the shape you have for your batch is (1, 1, 256, 256, 128)

Regards, Fausto

On Mar 15, 2017, at 12:05 PM, sagarax009 notifications@github.com wrote:

hi @faustomilletari https://github.com/faustomilletari I made changes in test.prototxt file with dimension 256,256,128

but I am getting below error

File "/net/linse8-sn/no_backup_01/d1202/VNet-master/VNet.py", line 195, in test net.blobs['data'].data[...] = btch ValueError: could not broadcast input array from shape (1,1,128,128,64) into shape (1,1,256,256,128)

Do I need to train model again by changing train.prototxt file too or am I missing something?

thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19#issuecomment-286790925, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvpfIaV6s9_Prd9E7f9aLaU-Zz3dDks5rmAxbgaJpZM4MbzzB.

[sagarhukkire]

Hi @faustomilletari can you please elaborate following point

2) you are feeding into the network data that is padded. the padding has to be done manually ( :( ) i think you can use numpy to do it without issues.

As of now I am passing entire image there is no padding as image itself is (293,176,429). if possible can You can send me any reference

with this example http://stackoverflow.com/questions/19349410/how-to-pad-with-zeros-a-tensor-along-some-axis-python understood that i have to define tensor then as per padded zeros I have to set the dimension in prototxt

And yes I am using test_noPooling_ResNet_cinque.prototxt only i just changed input size and reshape parameters in prototxt

[sagarhukkire]

Thanks now i realised mistake i will try and let you know

Thanks and Regards Sagar Hukkire On Mar 15, 2017 5:09 PM, "Fausto Milletari" notifications@github.com wrote:

No, I don’t think so. I think you might need to be sure that:

1) you are using the correct prototxt for testing -- “test” prototxt 2) you are feeding into the network data that is padded. the padding has to be done manually ( :( ) i think you can use numpy to do it without issues.

check with btch.shape if the shape you have for your batch is (1, 1, 256, 256, 128)

Regards, Fausto

On Mar 15, 2017, at 12:05 PM, sagarax009 notifications@github.com wrote:

hi @faustomilletari https://github.com/faustomilletari I made changes in test.prototxt file with dimension 256,256,128

but I am getting below error

File "/net/linse8-sn/no_backup_01/d1202/VNet-master/VNet.py", line 195, in test net.blobs['data'].data[...] = btch ValueError: could not broadcast input array from shape (1,1,128,128,64) into shape (1,1,256,256,128)

Do I need to train model again by changing train.prototxt file too or am I missing something?

thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ faustomilletari/VNet/issues/19#issuecomment-286790925, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvpfIaV6s9_ Prd9E7f9aLaU-Zz3dDks5rmAxbgaJpZM4MbzzB.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19#issuecomment-286792112, or mute the thread https://github.com/notifications/unsubscribe-auth/ATFxy00Bg_RFPx_zOsqeT6cXiClSmGvHks5rmA0xgaJpZM4MbzzB .

[sagarhukkire]

I can see more segmentation is appearing even I have to try with padding zeros.

[faustomilletari]

So i think the main suspects here are some issues with the padding used in the convolutions of the network. In the convolutional layers of the network, the convolutions are done such that the resulting filtered image has the SAME size as the previous one. Padding voxels with zero value are added to the images prior to convolution.

this thing of doing or not doing padding in the convolutions network is a bit controversial. In U-Net the authors are applying no padding and they need to process the volume tile-wise, in an overlapping tile manner. Other architectures, which have receptive fields much bigger than the initial images CANNOT produce results without applying padding during convolutions! This is the case for all res-nets.

I never experienced problems due to padding during convolutions in V-Net, I suggest you to try -- just for fun -- to segment the SAME images you used for training and observe if they are 'cut' like these ones. See what you get in that case.

In general I think that I didn't observe many problems in my application of V-Net because I was operating mostly with volumes that were already zero padded, in some cases with quite a bit of pixels, and also because the prostate was occurring around the center of the volume.

One thing that will solve your problem for sure -- if the cause of this behavior is the padding used during convolution -- is to use padded images also during training. The padding you need to apply will be quite a bit in this case, according to your images. This will increase your memory requirements!

another problem connected with this one could be represented by our way of doing pooling (here we do it through convolutions). In this case I could suggest to you to switch back to usual max pooling.

finally, another unlikely problem, could be connected with some weird things happening to the memory locations from which you read the output of the network. Namely a bug in the implementation. (very difficult bug to find...)

[sagarhukkire]

Hi Thanks for your message.

Here are ideas which come to me

1. I will try first with padding zeros If results are coming then I will make padding for training too

2) or I will pass just VOI ( volume of interest ) which will fit the predefined voxel size of Network

And beforehand i wouldlike to thank you for such awesome model which I can figure out any information in short notice .

I will keep you posted the next results soon

Thanks and Regards Sagar Hukkire On Mar 15, 2017 9:34 PM, "Fausto Milletari" notifications@github.com wrote:

So i think the main suspects here are some issues with the padding used in the convolutions of the network. In the convolutional layers of the network, the convolutions are done such that the resulting filtered image has the SAME size as the previous one. Padding voxels with zero value are added to the images prior to convolution.

this thing of doing or not doing padding in the convolutions network is a bit controversial. In U-Net the authors are applying no padding and they need to process the volume tile-wise, in an overlapping tile manner. Other architectures, which have receptive fields much bigger than the initial images CANNOT produce results without applying padding during convolutions! This is the case for all res-nets.

I never experienced problems due to padding during convolutions in V-Net, I suggest you to try -- just for fun -- to segment the SAME images you used for training and observe if they are 'cut' like these ones. See what you get in that case.

In general I think that I didn't observe many problems in my application of V-Net because I was operating mostly with volumes that were already zero padded, in some cases with quite a bit of pixels, and also because the prostate was occurring around the center of the volume.

One thing that will solve your problem for sure -- if the cause of this behavior is the padding used during convolution -- is to use padded images also during training. The padding you need to apply will be quite a bit in this case, according to your images. This will increase your memory requirements!

another problem connected with this one could be represented by our way of doing pooling (here we do it through convolutions). In this case I could suggest to you to switch back to usual max pooling.

finally, another unlikely problem, could be connected with some weird things happening to the memory locations from which you read the output of the network. Namely a bug in the implementation. (very difficult bug to find...)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19#issuecomment-286870331, or mute the thread https://github.com/notifications/unsubscribe-auth/ATFxy2-J3petsjENZT0N4RpTzeNYiM7Kks5rmEtfgaJpZM4MbzzB .

[sagarhukkire]

@faustomilletari

I passed zero padded (50 zeros along X and Y axis ) data to network , result looks little bit nice but still CUT is there.

P.S : I have trained model for 13500 iterations ,do i need to increase iteration or anything I need to change

[sagarhukkire]

@gattia

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

[faustomilletari]

I know what's happening to you.

Padding during convolutions has nothing going to do with this at all!!

The fault is the data Manger. But thank god we have a way to find out if this is really the case.

The data manager loads your data and resizes it to 256,256,128 (or anything else you choose in your configuration) at a resolution of 1, 1, 1.5 mm per pixels most probably, depending on your configuration. There are two settings in the configuration in main.py for that they have names that resemble "volume resolution" and "volume size".

When you feed a volume in, no matter what size it is, it gets resemble at the spatial resolution you choose in the configuration (this could result in a huge volume much bigger than 256,256,128) and then it gets CROPPED to match the size you requested (256,256,128).

Im pretty sure that the parts of images that you don't see are cropped out by this process.

When the results are written back the cropping and resempling is reversed! So you will never know!!

The trick you can do to see what is going on is to take the batch right before it gets into the network. Install and import the python module nibabel, make your batch 3D by suppressing singleton dimensions, and ask nibabel to dump it to disk in nii format (this super easy with nibabel as opposed to simpleitk). Then you open it and you look at what your network REALLY sees!!

I'm sure this is the problem. 99.9% sure, really!!

Fausto Milletarì Sent from my iPhone

On 16 Mar 2017, at 08:09, sagarax009 notifications@github.com wrote:

@gattia

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[faustomilletari]

As a rule in deep learning.

Always look at what you are giving as input right when you feed the input to the net. And always look at the value types and intervals.

Fausto Milletarì Sent from my iPhone

On 16 Mar 2017, at 08:09, sagarax009 notifications@github.com wrote:

@gattia

The dimension is (293,176,429), yes in all segmentation I am getting cropped data.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sagarhukkire]

@faustomilletari

indeed you are right the image passed to network is getting cropped so the final label coming.

[faustomilletari]

Yuppy!! I’m very happy to see the that was the problem. I would have been concerned if it would have been something else.

So… it’s just needed that you work at a lower resolution.

Can I ask you what you are working on? Do you want to reach me on my personal email to discuss maybe? My personal email is my name dot my surname at gmail dot com

(discussing it here is better though. i encourage you to discuss here! also i encourage you to put your code on GitHub and also to make a pull request with some code to produce a warning if important (non zero) portions of a volume get cropped away by the data manager)

Regards,

Fausto

On Mar 16, 2017, at 12:10 PM, sagarax009 notifications@github.com wrote:

@faustomilletari https://github.com/faustomilletari indeed you are right the image passed to network is getting cropped so the final label coming.

https://cloud.githubusercontent.com/assets/20017611/24006148/348b5e06-0a6b-11e7-982a-3594fb788e61.png — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/faustomilletari/VNet/issues/19#issuecomment-287107556, or mute the thread https://github.com/notifications/unsubscribe-auth/AMtsvpcCAK5HzaFK6chMLNjomeQetJ_3ks5rmV7mgaJpZM4MbzzB.

[sagarhukkire]

@faustomilletari

Got it bingo...you are amazing man !!! many thanks ,hope to see you once!!

The solution is 1) Do zero padding to X and Y axis to training images and labels 2) Do the lower resolution at first

:+1: I have to start training again ..but this time I am sure I will hit the result

[sagarhukkire]

after padding input data and lower resolution solves the issue