How many data can we use in order to training?

[tphankr]

Thank you for your code. I saw in the Data folder, have only two "image_A.nii" and "image_B.nii". Allows me have question:

1) With Image Registration problem, we only use two image for training step ( python Train_cLapIRN.py). Can we use a lot data for training step?. For example, we use 100 image .nii for the training step. Thank you.

[cwmok]

In short, yes, you can and you should train c-LapIRN with a lot of data.

The "image_A" and "image_B" are attached for demonstrating the proposed method. The provided pretrain model has trained with the OASIS dataset. In fact, you should train your model with as much data as possible (to increase the generalizability of your model). Depending on the application, i.e. inter-/intra-patient registration, you may need to modify the class Dataset_epoch in Functions.py for training.

[tphankr]

In short, yes, you can and you should train c-LapIRN with a lot of data.

The "image_A" and "image_B" are attached for demonstrating the proposed method. The provided pretrain model has trained with the OASIS dataset. In fact, you should train your model with as much data as possible (to increase the generalizability of your model). Depending on the application, i.e. inter-/intra-patient registration, you may need to modify the class Dataset_epoch in Functions.py for training.

Thank you @cwmok . I am trying do that.

• download the OASIS dataset
• do training step.

I will respond the result.

[cwmok]

@tphankr Please remember to preprocess the data before you use it in training. Or you may download the preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md Make sure your input data has an intensity range within [0, 1], similar to the attached "image_A" and "image_B".

Looking forward to seeing your result! :)

[tphankr]

@tphankr Please remember to preprocess the data before you use it in training. Or you may download the preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md Make sure your input data has an intensity range within [0, 1], similar to the attached "image_A" and "image_B".

Looking forward to seeing your result! :)

Thank you for your advises The first step, I will download "the preprocessed OASIS from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md." Thank you.

[tphankr]

@tphankr Please remember to preprocess the data before you use it in training. Or you may download the preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md Make sure your input data has an intensity range within [0, 1], similar to the attached "image_A" and "image_B". Looking forward to seeing your result! :)

Thank you for your advises The first step, I will download "the preprocessed OASIS from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md." Thank you.

Thank you @cwmok and everyone. I choose the way download the data that avoid preprocess the data into [0,1]. I had the data but I can not traing. Please help me some next step, I am going to learn hard. Thank you @cwmok and everyone (@adalca)

I downloaded the OASIS from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md.". And it includes : + neurite-oasis.2d.v1.0 folder ( 2D**)

• neurite-oasis.v1.0 folder (3**) Each folder has 457 folder that has .nii.gz file. I take a pictures for (3**)to make it easier for us to imagine.

And then I typed python Train_cLapIRN.py --datapath ../neurite-oasis.v1.0/OASIS_OAS1_0001_MR1/ or I also tried to typed for (2D**) python Train_cLapIRN.py --datapath ../neurite-oasis.2d.v1.0/OASIS_OAS1_0001_MR1/

They had the same error "valueError: num_samples should be a positive integer value, but got num_samples=0". I understand, obviously the OASIS_OAS1_0001_MR1 folder had aligned_norm.nii.gz, aligned_orig.nii.gz, orig.nii.gz.....Why did I type python Train_cLapIRN.py --datapath ../neurite-oasis.v1.0/OASIS_OAS1_0001_MR1/ have the error. Please guide me. Thank you @cwmok and everyone.

Thank you.

[cwmok]

@tphankr I guess the error is from the naming convention. In our code, (line 78, 184 and 296 in Train_cLapIRN.py) names = sorted(glob.glob(datapath + '/*.nii')), we select all .nii files within the folder datapath. Change "*.nii" to "*.nii.gz" may solve your problem.

[tphankr]

Thank you very much for your help. I solved the last problem when follow your advises on your above comment (line 78, 184 and 296 in Train_cLapIRN.py) names = sorted(glob.glob(datapath + '/*.nii'))

1) Now I faced the error and do not fix them yet: "RuntimeError: Sizes of tensors must match except in dimension 2. Got 160 and 256." I thought all day but can not solution this problem. Please help, thank you.

  ***MY SOLUTION BUT do not fix them yet. I changed 160 to 256 at lines 396 to 398 but at this time have the same error: "RuntimeError: Sizes of tensors must match except in dimension 2. Got 256 and 160"

2) Can you help me understand more about the preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md. Will we traing with all 12 files ( aligned_norm.nii.gz; aligned_orig.nii.gz; ....orig.nii.gz; ......) or training only with the files ( norm.nii.gz; orig.nii.gz) ? as below following:

I also recall the order folders in preprocessed OASIS dataset from @adalca, in order to you can imagine the data:

With OASIS_OAS1_0001_MR1 folder: And with OASIS_OAS1_0002_MR1 folder:

Thank you @cwmok .

[cwmok]

@tphankr You don't need all the ".nii.gz" data in each case to train the model. You only need the "aligned_norm.nii.gz" from each case in order to train your own model. I will show you how to train with your dataset step by step.

Step 1: Change --datapath to "../neurite-oasis.v1.0". Step 2: Change names = sorted(glob.glob(datapath + '/*.nii')) in lines 78, 184 and 296 to names = sorted(glob.glob(datapath + '/*/aligned_norm.nii.gz')). This will create a list of path with "aligned_norm.nii.gz" in each case. Step 3: Change Dataset_epoch(names, norm=False) to Dataset_epoch(names, norm=True) in lines 93, 198 and 313. This will enable the min-max normalization in the data generator.

This should be able to train the model without breaking the file structure of the preprocessed dataset.

For the details of the preprocessed dataset, you may refer to the description in https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md.

Look forward to seeing your results.

[tphankr]

Thank you very much for your helps. I am going try learning hard. I did the training step, thank you @cwmok. Your code is trained with dataset from preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md. The solution is written at bottom. Allows me have share some my opinion. Thank you very much. Question1) We only need to do with the "aligned_norm.nii.gz" file in each foder (OASIS_OAS1_0001_MR1, OASIS_OAS1_0002_MR1, ....; all file is 457 folder) in order to have losses in Model/Stage folder and then we forword to test step, as following: ( because I saw still many files we do not use it yet. for example (aligned_orig.nii.gz; aligned_seg4.nii.gz; ....). . My tartget, now: .

                       ################ The Solution      ##################

Solution for the code training : 1) I followed your steps carefully. 2) And need change the shape at from the line 382 to the line 384 because the shape of aligned_norm.nii.gz in each folder that downloads from preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md is 160 x 192 x 224. #################################################

Thank you @cwmok. Have a nice day to you.

[tphankr]

Thank you @cwmok. The code was training on three day ago, with preprocessed OASIS dataset from @adalca at https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md. Thank you @cwmok. It was training three day but now the code still hasn't finished runing. Allow me ask, when do the code train finishly? I want to change to the test step in order to have the result.

• The preprocessed OASIS dataset has 457 folder. In each folder has many file ( aligned_norm.nii.gz, aligned_orig.nii.gz, ..., aligned_orig...). The code is training with aligned_norm.nii.gz file in each folder, as you know.
  • My computer: + GPU ( 1080 ti, 12GB)
  • the "start channel" is 4 ( because memory is outed so I changed to 4)

Thank you @cwmok.

[cwmok]

@tphankr The training will complete when the number of steps reaches "--iteration_lvl3". By default, --iteration_lvl3 is set to 60000. The training speed is subject to many factors, e.g., the spatial resolution of your data, model complexity, number of iteration, hardware and data complexity etc. In our experiment, we centre crop the data and use the default parameters of the program. One complete training should take ~28 hours with an Nvidia Titan RTX GPU and Intel Core (i7-4790) CPU.

[tphankr]

@tphankr The training will complete when the number of steps reaches "--iteration_lvl3". By default, --iteration_lvl3 is set to 60000. The training speed is subject to many factors, e.g., the spatial resolution of your data, model complexity, number of iteration, hardware and data complexity etc. In our experiment, we centre crop the data and use the default parameters of the program. One complete training should take ~28 hours with an Nvidia Titan RTX GPU and Intel Core (i7-4790) CPU.

Thank you @cwmok. I saw it in the table 1,  I will try to learn hard to have the base-line results.
I am waiting for the training step completely and change to the next test step, now 

My computer has been training for almost 3days. My computer: + Nvidia GeForce GTX 1080 Ti, 12GB andn Intel Core ( i9-10900K).

Allows me ask: I saw in the "Test_cLapIRN.py", I did not see the coding compute the DSC, std(|J| phi), %std(|J phi|) of Table 1. Where can I find the files in order to computer these coefficients?

The test file:

Thank you, @cwmok

[tphankr]

Thank you @cwmok . Yeah...the traning step have just completed with around 48 hours on my computer. Please help me the next step. I am wondering the remaining what files in each folder of preprocessed OASIS dataset, 1) which files should be used for the test step because have many files in each folder ?

  I am also trying the testing step with two files in the OASIS_OAS1_0001_MR1 ( fixed: orig.nii.gz; moving: norm.nii.gz), as below following but I am not sure true. I changed the link of "fixed"  and "moving" in the Test_cLapIRN.py, as following: 

And I typed:

Thank you, @cwmok

[cwmok]

@tphankr

Where can I find the files in order to compute these coefficients?

We didn't release the evaluation code. You may need to implement it by yourself. For your reference, Dice score at https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient and Jacobian matrix at https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant.

Which files should be used for the test step because have many files in each folder?

It seems you are confusing with the concept of image registration and the OASIS dataset. For image registration, I suggest you revisit the great tutorial of VoxelMorph in https://colab.research.google.com/drive/1WiqyF7dCdnNBIANEY80Pxw_mVz4fyV-S?usp=sharing. For the OASIS, as I mentioned earlier, the details are well-described in https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md. In short, the preprocessed OASIS dataset provided >400 cases and are separated by different folders. Each folder contains raw image in the scanner and template space. Throughout the training, your trained model already learnt how to register images in template space and your goal is to register all images in template space. For the slice_*.nii.gz files, it contains the anatomical segmentation maps of each case, which can be used to evaluate the trained model.

Also, if you are confusing about the experiment/the concept of deep learning-based image registration method, please also check out the pioneering work on image registration in https://arxiv.org/abs/1802.02604.

[tphankr]

Thank you @cwmok. Thank you for your response well. I try to read your comment again and again ( papers,....) and follow your advice.

Thank you @cwmok .

[tphankr]

Which files should be used for the test step because have many files in each folder?

It seems you are confusing with the concept of image registration and the OASIS dataset. For image registration, I suggest you revisit the great tutorial of VoxelMorph in https://colab.research.google.com/drive/1WiqyF7dCdnNBIANEY80Pxw_mVz4fyV-S?usp=sharing. For the OASIS, as I mentioned earlier, the details are well-described in https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md. In short, the preprocessed OASIS dataset provided >400 cases and are separated by different folders. Each folder contains raw image in the scanner and template space. Throughout the training, your trained model already learnt how to register images in template space and your goal is to register all images in template space. For the slice_*.nii.gz files, it contains the anatomical segmentation maps of each case, which can be used to evaluate the trained model.

Also, if you are confusing about the experiment/the concept of deep learning-based image registration method, please also check out the pioneering work on image registration in https://arxiv.org/abs/1802.02604.

Thank you @cwmok. These weeks, I followed your advice ( tutorial of VoxeMorph and read the paper). I knew, when we do the test step, we only use two image ( the Moving image and the Fixed image). For example, I typed "python Test_cLapIRN.py --reg_input 0.41" and had the results, as following:

In this time, I had the image registration results of norm.nii.gz (fixed image) and norm.nii.gz ( moving image) with the first folder of OASIS dataset. I will do the test step with orthers folder, right? .

Please allow me ask: Question1: I had the image registration results of norm.nii.gz (fixed image) and norm.nii.gz ( moving image) with the first folder of OASIS dataset. I will do the test step with orthers folder, right?

Question2: Now, I have a born scan dataset like this ( 2D image with .tif). Does your code use to train with 2D image? Can you guide me the code in order to do with a 2D image (.tif)?

Thank you @cwmok .

[cwmok]

@tphankr

Question1: I had the image registration results of norm.nii.gz (fixed image) and norm.nii.gz ( moving image) with the first folder of OASIS dataset. I will do the test step with orthers folder, right?

No. In our inter-patient brain registration experiments, we register the norm.nii.gz in each case to a selected atlas. For example, Fixed scan = OASIS_OAS1_0001_MR1/norm.nii.gz, moving scan = OASIS_OAS1_0002_MR1/norm.nii.gz. For details, please refer to our paper. In your screenshot, you are registering the same brain scan in the original and template space, which is not what we did in our paper.

Question2: Now, I have a born scan dataset like this ( 2D image with .tif). Does your code use to train with 2D image? Can you guide me the code in order to do with a 2D image (.tif)?

Currently, this code supports 3D image registration only. If you want to apply it on 2D image registration, you need to change some codes such as the deformation grid size, the data size permutation, conv3D -> conv2D etc, which may not be trivial if you are new to Pytorch and image registration.

[tphankr]

@tphankr

Question1: I had the image registration results of norm.nii.gz (fixed image) and norm.nii.gz ( moving image) with the first folder of OASIS dataset. I will do the test step with orthers folder, right?

No. In our inter-patient brain registration experiments, we register the norm.nii.gz in each case to a selected atlas. For example, Fixed scan = OASIS_OAS1_0001_MR1/norm.nii.gz, moving scan = OASIS_OAS1_0002_MR1/norm.nii.gz. For details, please refer to our paper. In your screenshot, you are registering the same brain scan in the original and template space, which is not what we did in our paper.

Question2: Now, I have a born scan dataset like this ( 2D image with .tif). Does your code use to train with 2D image? Can you guide me the code in order to do with a 2D image (.tif)?

Currently, this code supports 3D image registration only. If you want to apply it on 2D image registration, you need to change some codes such as the deformation grid size, the data size permutation, conv3D -> conv2D etc, which may not be trivial if you are new to Pytorch and image registration.

Thank you very much @cwmok. The next, I am goning to read your paper carefully.

I will try do learn hard. Thank you @cwmok

[tphankr]

Dear @cwmok ,

I have been reading your paper. The last time, you trained already and have some results to I understand. Allow me ask, Do we need the label when run the code? For example: Allows me ask and help me. I downloaded the dataset that have the masks ( labels) folder and input folder from the Learn2Reg challenge. I download and have like this:

How to apply in your code? please help me. Thank you very much.

@ I am following you in order to learn so I don't use these results to submit to this challenge for this year. I only want to learn about the Image Registration problem. Thank you @cwmok .

[cwmok]

@tphankr

Allow me ask, Do we need the label when run the code?

This is optional. The mask can be used in the loss function as stated in A-LapIRN. or simply drop it away.

How to apply in your code? please help me. Thank you very much.

If you want to apply c-LapIRN to intra-patient lung registration, you may want to read our workshop paper A-LapIRN and follow these steps:

1. (Optional) Downsample the data. In the challenge, they only need solutions with half-resolution. This also helps to reduce the training complexity.
2. Create your data generator for intra-patient registration. In contrast to inter-patient brain registration, this dataset requires registration between the exhale and inhale lung scans from the same patient. 2.1 Preprocess the data with the min-max normalization within the custom data generator and make sure that the background intensity is equal to 0. (!!! very important)
3. Data augmentation. As the number of training data is limited, you need to augment your training data in order to improve the generalize ability of the model.

Currently, I have a hectic schedule with the journal paper and other projects. I may not be available to give you a concrete example of how to apply c-LapIRN to this dataset. Yet, I will definitely share the tips and tricks if I have time to participate in this year challenge.

Stay tuned.

[tphankr]

Thank you very much @cwmok with your useful advises. I will study your advice carefully and study day by day to learn and do it. I know I'm too weak to understand your advice but I'll try. My plans: + follow your workshop paper A-LapIRN and your advises.

• and besides I aslo find the code that have apply intra-patient lung registration to I follow.

Thank you @cwmok.

[tphankr]

@tphankr

Allow me ask, Do we need the label when run the code?

This is optional. The mask can be used in the loss function as stated in A-LapIRN. or simply drop it away.

How to apply in your code? please help me. Thank you very much.

If you want to apply c-LapIRN to intra-patient lung registration, you may want to read our workshop paper A-LapIRN and follow these steps:

1. (Optional) Downsample the data. In the challenge, they only need solutions with half-resolution. This also helps to reduce the training complexity.
2. Create your data generator for intra-patient registration. In contrast to inter-patient brain registration, this dataset requires registration between the exhale and inhale lung scans from the same patient. 2.1 Preprocess the data with the min-max normalization within the custom data generator and make sure that the background intensity is equal to 0. (!!! very important)
3. Data augmentation. As the number of training data is limited, you need to augment your training data in order to improve the generalize ability of the model.

Hello @cwmok,

1. The last time, I download the data: Can you guide me a link to follow, how do we do the 2.1 step? "2.1 Preprocess the data with the min-max normalization within the custom data generator and make sure that the background intensity is equal to 0. (!!! very important)" I searched about the min-max normalization but I also don't how do I do this step. Thank you very much.

[cwmok]

@tphankr

I searched about the min-max normalization but I also don't how do I do this step. Thank you very much.

Here is it.

def imgnorm(img):
    max_v = np.max(img)
    min_v = np.min(img)
    norm_img = (img - min_v) / (max_v - min_v)
    return norm_img

Also, please also check out the windowing operation at this link.

[tphankr]

@tphankr

I searched about the min-max normalization but I also don't how do I do this step. Thank you very much.

Here is it.

def imgnorm(img):
    max_v = np.max(img)
    min_v = np.min(img)
    norm_img = (img - min_v) / (max_v - min_v)
    return norm_img

Also, please also check out the windowing operation at this link.

Thank you very much @cwmok. I will do immedietely.

[tphankr]

@tphankr

I searched about the min-max normalization but I also don't how do I do this step. Thank you very much.

Here is it.

def imgnorm(img):
    max_v = np.max(img)
    min_v = np.min(img)
    norm_img = (img - min_v) / (max_v - min_v)
    return norm_img

Also, please also check out the windowing operation at this link.

Thank you @cwmok, All this week, I trained your code with Train_LapIRN_diff.py file with preprocess the data with the min-max normalization of task 3 of Learn2Reg ( sorry the last time, I said wrongly task2). Now, I had bad results, as following: +In order to do the preprocess the data with min-max normalization ( "2.1 Preprocess the data with the min-max normalization within the custom data generator and make sure that the background intensity is equal to 0. (!!! very important)"). I did this idea by setting norm=True on the line 139 and 163 in the class Dataset_epoch and class Predict_dataset of the Function.py file, right? Now, I have the flow and moving result of Fixed and Moving image as following, I do not know how is the bad results. Please guide me more ( I also tried trainly with norm=False but the same ). Thank you @cwmok.

Thank you @cwmok.

[cwmok]

Hi @tphankr,

This task is a more difficult and challenging task due to its limited training data and large displacement. I haven't tried the diffeomorphic version of LapIRN on this task. All the details about how to ace this task can be found in my workshop paper "Large deformation image registration with anatomy-aware Laplacian pyramid networks"

Here are some tips for you to avoid the pitfalls in this task:

1. I downsample the training data to half-resolution (96, 80, 128) in order to speed up the training time and increase the number of feature maps (start_channel=28).
2. You will need larger smoothness regularization, i.e., --smooth set to [2, 4].
3. You forgot the CT windowing. It helps to make sure that the background intensity is equal to 0 after min-max normalization. Print the preprocessed image out using Matplotlib to double-check your preprocessed image.
4. Data augmentation or supervised loss (e.g., Dice loss of anatomical structure, see [VoxelMorph])(https://arxiv.org/abs/1809.05231) are necessary for training on a small dataset.
5. Use Train_LapIRN_disp.py instead of Train_LapIRN_diff.py. The method parameterized with displacement field is simpler and more stable in general.

To reiterate, this task is difficult compared to brain registration. You may want to try our method on brain registration first before trying it on this task.

[tphankr]

Thank you for your nice reply. Yes, I will start again with brain registration (the preprocessed OASIS from @adalca at .") and then do this task. Now I continue again...

I tried to learn CT windowing software. Is it software that we upload images and then the background intensity output is downloaded equal to 0? I will try again and again because now the upload file is errored the first nii.gz file of task 3 of Learn2Reg 2020. Thank you @cwmok.

Thank you @cwmok.

[tphankr]

Dear @cwmok, Can you guide me something more about the step 3? Becasue, I lost my way in this step. "You forgot the CT windowing. It helps to make sure that the background intensity is equal to 0 after min-max normalization. Print the preprocessed image out using Matplotlib to double-check your preprocessed image."

I still have bad result for task3.

Thank you, @cwmok .

[cwmok]

@tphankr

I decided to release a prototype using the preprocessed OASIS dataset so that everyone new to image registration can easily build their own image registration method using LapIRN. I will release it very soon.

For the CT windowing technique, the goal is to increase the contrast of the targeting anatomical structures. It can be achieved using the NumPy clip function by setting an appropriate intensity range.

There are too many pitfalls in image registration. I recommend you try the classic VoxelMorph registration method to get you familiar with the image registration problem. Then, you might follow the coming prototype to modify the code to adapt to any tasks.

[tphankr]

@cwmok, Thank you very much @cwmok. I continous...

[cwmok]

Hi @tphankr,

I have updated the code and added an example using the preprocessed OASIS dataset without cropping/resize. You may want to check out the section "(Example) Training on the preprocessed OASIS dataset without cropping" in the "README.md" file.

[tphankr]

Hi @tphankr,

I have updated the code and added an example using the preprocessed OASIS dataset without cropping/resize. You may want to check out the section "(Example) Training on the preprocessed OASIS dataset without cropping" in the "README.md" file.

Thank you very much, @cwmok . I am studying and learning from your comments.

Thanks.