search

fepegar / torchio

Medical imaging toolkit for deep learning
http://www.torchio.org
Apache License 2.0
2.01k stars 233 forks source link

Add support for 4D images #238

Closed fepegar closed 3 years ago

fepegar commented 4 years ago

🚀 Feature

Add support to read, write, sample and transform 4D images.

Motivation

It would be convenient to support 4D images out of the box, as tensors of shape (C, S1, S2, S3) where C is the number of channels and SX are spatial dimensions (S1 = 1 for 2D images).

Examples of what the 4th dimension might encode

Gradient directions

Time

Modalities or MRI sequences

EM frequency bands

Labels

Pitch

Support of 4D images in:

Alternatives

Pass the channels independently in different instances of Image.

Additional context

Considerations

torchio.io

Possible image shapes when reading

How do we know if 2D or 3D when there are 3 dimensions in the read data? Lookup table of file extensions?

2D
3D

Transforms

Consider whether to apply the same or different transformations to each channel of each image, as pointed out by @romainVala in https://github.com/fepegar/torchio/issues/238#issuecomment-661708748.

Related issues

romainVala commented 4 years ago

Great

There is no obvious difficulties for read write and sample since it is just adding multi-channel ...

Here are just a few thought for what to do with transform

For spatial transform : one would like to have the same transfo apply to each channel. Either a for loop on the channel, on directly apply on 4D, (which is strangely slower for resample as shown in #213 )

Although in the case of diffusion one may be interested, to produce different elastic transform (as there are small non linear deformation between diffusion volume) ... so may be the same option as for intensity ?

For intensity transform: we should have the choice (with a user define parameter) between applying the same transform for each channel or a different one

As an example for Motion, with diffusion or fMRI data one expect to have a different motion for each volume, but with a multi echo sequence, we expect the same motion for each volume

It seem logical that for random_skipke we use a different one, but for random_bias the same for each channel ... but I suggest to let the user decide

this may be quite a lot of code change in each transform ...

romainVala commented 4 years ago

Actually the same choice of behaviour should also be possible when multiple images are load in the subject sample

For instance in randomAffine the same affine parameter is applied to all images (one could want a different one too) For RandomNoise it is the opposite ...

It would be great and flexible if this can also be a user choice (as input param of the transform)

Because the choice of loading several 3D volume or one 4D volume will mainly be done depending of how the file are stored, but in this is 2 different internal representation of the same data ... so It make sens to have the same behavior

fepegar commented 4 years ago

this may be quite a lot of code change in each transform ...

Yes, this won't be easy! We need to plan this feature carefully.

fepegar commented 3 years ago

Would it be a good idea to add new subclasses of Image? That would help interpret the shape of the read image. Or maybe a kwarg in Image specifying the number of spatial dimensions (2 or 3).

fepegar commented 3 years ago

I've done most of the work. The parameters are computed per-channel, but I think it's good to merge that for now and we'll add support to have the choice later.

@GFabien after merging this, could you refactor RandomLabelsToImage? I think the code will get much more elegant.

GFabien commented 3 years ago

@GFabien after merging this, could you refactor RandomLabelsToImage? I think the code will get much more elegant.

I agree. If I do this I may add a kwarg to choose the channels used to create the new image because I really like the modularity brought by the fact of having the labels as different keys in the sample. For example, in some of the models I'm currently running I use a OneOf between two RandomLabelsToImage, one that includes extra brain mask and samples a gaussian in these regions and the other one that excludes them and takes the values from the original T1 image. Without such a kwarg I would need to create two different label volumes...

fepegar commented 3 years ago

If I do this I may add a kwarg to choose the channels used to create the new image because I really like the modularity brought by the fact of having the labels as different keys in the sample.

Sounds good!

fepegar commented 3 years ago

FYI, most transforms seem ok: https://drive.google.com/file/d/1Gc8kzwKQR-bYA_ifqTnA6v0N5J_hPUeO/view?usp=sharing

romainVala commented 3 years ago

Sorry If I misunderstand, but I only had a quick look at the code, (i am in holiday ...) I wonder what is the choice you have made, for all transform when you have 4D image If I correctly understand, for random_biasfield, you apply a different bias field to each 4D image ? Is that correct ? I thing this should be a user choice of the transform whether you apply the same bias field to all 4D images or a different one

(from a physical point of view it makes sense to apply the same bias field to all 4D images, as it is what happen during the acquisition (if the subject is no moving too much into the coil)

fepegar commented 3 years ago

Salut Romain,

I wonder what is the choice you have made, for all transform when you have 4D image

For now, I haven't made any choice. I just adapted Image and the transforms to take 4D images.

If I correctly understand, for random_biasfield, you apply a different bias field to each 4D image ? Is that correct ?

I think that's not what happens in the current implementation. I wrote to visualize the 4D transforms, look: https://colab.research.google.com/drive/1Gc8kzwKQR-bYA_ifqTnA6v0N5J_hPUeO#scrollTo=Dy2v05LPVCvA&line=2&uniqifier=1

I also think it should be the user's choice, but I won't have time to work on that anytime soon. Contributions are welcome, this shouldn't be difficult to implement for most of the transforms.

fepegar commented 3 years ago

Enjoy your holidays!

meghbhalerao commented 3 years ago

Hi all, Thank you so much for adding 4D image support! I was wondering if there is support for directly converting a 3D multi class label map (for example in BraTS you have pixel values as 0,1,2,4) into a one-hot encoded label map which is used for model training in semantic segmentation? As in if I just instantiate an object of the class Subject with torchio.LABEL as some .nii.gz file which has a BraTS mask, would it convert this mask into a one-hot mask when I read torchio.LABEL while iterating through the DataLoader? Please let me know if something is not clear. Thank you

fepegar commented 3 years ago

Hi, @meghbhalerao.

I was wondering if there is support for directly converting a 3D multi class label map (for example in BraTS you have pixel values as 0,1,2,4) into a one-hot encoded label map which is used for model training in semantic segmentation? As in if I just instantiate an object of the class Subject with torchio.LABEL as some .nii.gz file which has a BraTS mask, would it convert this mask into a one-hot mask when I read torchio.LABEL while iterating through the DataLoader?

There's nothing one-hot-encoding-related in the library, but we could add it if necessary. You can use torch.nn.functional.one_hot.

Note that you can now use the LabelMap class and forget about torchio.LABEL.

meghbhalerao commented 3 years ago

Thanks for the clarification @fepegar!