Applying Transform Chain to Geometric Objects

[vikashg]

Is your feature request related to a problem? Please describe. Sometimes we need to apply transforms to geometric objects (point clouds, lines, curves, and meshes), particularly in radiotherapy planning and object detection.

Describe the solution you'd like A good solution would be a branch core library that can handle all the geometric transforms while accounting for the image metadata like image spacing and size. A geometric transform chain should be callable in a similar way as we call the regular image transforms. Something like

from monai.transforms.geometric import LoadPoints, RotatePoints

In an ideal case when we have an image annotation pair. We apply the same set of transformation to both objects. But for geometric objects there should be some way of passing messages to the annotations transform chain. For example you are rotating images and annotations. You can apply the rotation to the annotation, but after rotation the size of the image changes in different axes and sometimes it is important that the geometric transform should know about these changes for accurately applying the transforms. So there should be a message passing mechanism between two transform chains. More relevance in discussion #4024

Describe alternatives you've considered I have written my own transforms to do all these things but it is a bit hacky and not standardized. I have my geometric labels from open-source tools like labelme. My code handles labelme annotations (in a json) but going in I think we will need to have support for STL, DICOM RT annotations.

Additional context If there are some people working on it, please let me know as I have been working on it and will like to contribute to this branch.

[rijobro]

We really need to make progress on this PR before adding in point transforms (which is definitely on our agenda).

[MMelQin]

Just to add that MONAI Deploy App SDK has added the STL generation from volume segmentation image, closed issue and PR.

For generating DICOM RTStruct instance, for use in radiotherapy planning, the contour(s) of segment(s) on each segmentation image slice need to be calculated, with and without smoothing applied. I had plans to implement this using cv2 (more like porting from a earlier impl), though a utility/transform in MONAI core is better and can be for more general use.

[Nic-Ma]

Hi @vikashg @MMelQin ,

Thanks for the feedback, I agree with the requirements for point transforms, we are working in this direction. About for your particular question to calculate the contour, we have a similar transform LabelToContour: https://github.com/Project-MONAI/MONAI/blob/dev/monai/transforms/post/array.py#L508 And you can find some usage example in the tutorial: https://github.com/Project-MONAI/tutorials/blob/master/modules/postprocessing_transforms.ipynb

Thanks.

[ericspod]

As we consider more and more non-image data the architecture we use to integrate those transforms has to be considered. I'm not sure having a separate section in the transforms for geometric transforms is the way to go, but instead these would be treated at the same level as image transforms. A transform such as RandAffine would be applicable to image and geometry data, so RandAffined would also, rather than having separate transforms which would have to share state somehow. How we represent geometry is another issue since this can be a point cloud in 2D/3D with many other types of associated matrices such as connectivity.

[vikashg]

HI @ericspod Yes what are the other nonimage data are you thinking of. The only thing other than the geometric object I can think of is Text and audio data which are a whole different ball game altogether. It is true that the RandAffine can handle point cloud data as it is just matrix multiplication. Maybe we should throw in some examples together like a proper workflow with image and geometric data together. I will think of some and push a notebook and see if we may need some wrappers over the core functions.

[vikashg]

As we consider more and more non-image data the architecture we use to integrate those transforms has to be considered. I'm not sure having a separate section in the transforms for geometric transforms is the way to go, but instead these would be treated at the same level as image transforms. A transform such as RandAffine would be applicable to image and geometry data, so RandAffined would also, rather than having separate transforms which would have to share state somehow. How we represent geometry is another issue since this can be a point cloud in 2D/3D with many other types of associated matrices such as connectivity.

So @ericspod, if I choose to apply a rotation matrix to a set of points say something like this

points= np.asarray([[2, 3], [4, 9]])
transforms = Affine(rotate_params=0.35, padding_mode="zeros")
print(transforms(points))

I do get the expected error about spatial size

File "GeometricTransfroms.py", line 112, in test_image_rotation
    print(transforms(points))
  File "/workspace/MONAI/monai/transforms/spatial/array.py", line 1799, in __call__
    sp_size = fall_back_tuple(self.spatial_size if spatial_size is None else spatial_size, img.shape[1:])
AttributeError: 'list' object has no attribute 'shape'

However, the shape do not make much sense when you want to apply the transforms to a set of points. If i give a spatial size which might be the same as the image domain, I get the following error

  File "GeometricTransfroms.py", line 114, in test_image_rotation
    print(transforms(points, spatial_size=spatial_size) )
  File "/workspace/MONAI/monai/transforms/spatial/array.py", line 1799, in __call__
    sp_size = fall_back_tuple(self.spatial_size if spatial_size is None else spatial_size, img.shape[1:])
AttributeError: 'list' object has no attribute 'shape'

Maybe if there is a hook to extract the transformation matrix we can do the matrix multiplications discussed during the meetings.

I think if I dig deeper, i could hack some code that works but as of now the implementation is not particularly straight forward. I tried to find some solutions and @dongyang0122 also said that they are working on it https://github.com/Project-MONAI/MONAI/discussions/4024#discussioncomment-2462787

[ericspod]

The error I get for

import numpy as np
from monai.transforms import Affine
points= np.asarray([[2, 3], [4, 9]])
transforms = Affine(rotate_params=0.35, padding_mode="zeros")
print(transforms(points))

relates to spatial shape because points are understood: ValueError: Unsupported spatial_dims: 1, available options are [2, 3]. The Affine class currently only transforms image data, what we would want to do is extend it to work with points as well, such that if the input has shape [2,N] or [3,N] for some array of points, it can apply the transform to the points treating them as coordinates in space rather than some weird 1D image. How we detect which sort of data we have is something I was discussing with Richard in relation to the MetaTensor idea of combining metadata with a tensor which would state the data type.

[jejon]

Is someone still working on this?

[ericspod]

Hi @jejon we do want to implement this but currently our priorities for the next release is on enhancing bundles and their usability for new and experienced MONAI users. We also want to focus on integrating the generative models project into the core repo. For the following release this is one feature that we all feel is important and likely will be a priority then.

[ericspod]

I'll mention again that we should be looking at how other libraries do transforms and to keep geometric neural networks in mind which is a major use cases for these sorts of transforms, eg.: https://pytorch-geometric.readthedocs.io

[Nic-Ma]

Hi @ericspod ,

Thanks for sharing the pytorch geometric project, very inspiring. Is it possible to develop some wrapper or adapter to use pytorch geometric transforms in MONAI transform chain? Something like: https://docs.monai.io/en/stable/transforms.html#torchvision.

Thanks.

[ericspod]

In theory we should be able to adapt these transforms to MONAI quite simply. The base transform is just another callable so I suppose we could use these with Lambda/Lambdad now as it is. Some of the transforms are quite simple like RandomFlip but rely on their own data types like Data which defines meshes or shapes. We'd need a wrapper to interoperate with our dictionary based transforms, but more importantly to be able to mix and match transforms for applying to image and mesh data at once. For example, we'd want a RandomFlip transform to apply the same operation to both image and point data somehow, and it's not immediately obvious how we could do this without some significant rewriting. Laziness and invertibility are both large flies in the ointment as well.

[atbenmurray]

All of our spatial transforms have the ability to produce matrices describing the transform already. They just have different code-paths for when they are operating lazily or not. When they aren't operating lazily, the transform is calculated but then just put in applied transforms as if it had been used to perform the transform. When they are operating lazily, it goes on the pending list to be applied subsequently. Strictly speaking, we could put in a code path to every transform that applies the matrix to geometric data structures like points and meshes, but I'd like to propose that we go a bit further.

I'm going to try to sell the refactoring I did on the lazy resampling branch that refactored all the spatial transforms to use the same code path for lazy and non-lazy routes again. If all spatial transforms always work in terms of matrices (or deformation grids) then all we need is the ability to apply a matrix or grid to the appropriate data structure adapted from https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/data/data.py#L469

A pure spatial transform can be found here. It's a bit long so I'll just post the link:

https://github.com/atbenmurray/MONAI/blob/9b2ad34e8d5916f1a0cd21733be8b0b0e943e416/monai/transforms/spatial/functional.py#L400

[KumoLiu]

Hi all, I investigated more about https://pytorch-geometric.readthedocs.io/ and found that it's not easy to be compatible with the use cases in MONAI. For "pytorch-geometric", there is no concept of image, it refers to the cartesian coordinate system, but for detection, the transforms of the data refer to the center of the image.

Here I give a simple example with image size (10, 10), box size(2, 4) and flip it with pytorch-geometric and MONAI.

import torch
from monai.apps.detection.transforms.array import FlipBox
import torch_geometric.data as data
import torch_geometric.transforms as T

# monai
boxes = torch.tensor([[1, 3, 3, 7]])
trans_monai = FlipBox(spatial_axis=0)
out_monai = trans_monai(boxes, (10, 10))

# geometric
pos = torch.tensor([[1, 3], [1,7], [3, 3], [3, 7]])
data = data.Data(pos=pos)
trans_geometric = T.RandomFlip(axis=0, p=1)
out_geometric = trans_geometric(data)

print(boxes) # tensor([[1, 3, 3, 7]])
print(out_monai) # tensor([[7, 3, 9, 7]])
print(out_geometric.pos) # tensor([[-1,  3], [-1,  7], [-3,  3], [-3,  7]])

The red line represents the original image, and the red dotted line represents the original bounding box. The green and blue colors represent the results after the MONAI and Geometric Flip, respectively. We can find that the blue box is outside of the image which may not meet our requirements.

In summary, I propose we integrate the existing box transforms into the core. We can think of the operation on the box as an operator to integrate into the existing transform. If there are no other concerns, I'll try to create a draft PR first. https://github.com/Project-MONAI/MONAI/blob/38ac573dcd76181ad5c8a9a3cbc294e7d2fdb80d/monai/apps/detection/transforms/array.py#L213