3D transform for detection task

[Kirayue]

Is your feature request related to a problem? Please describe. Is there any way or will MONAI support 3-D transform for detection task, i.e. transform cube coordinate according to the image transform? Or how can we get an affine matrix we can use to get transformed coordinate?

Describe the solution you'd like Any solution is fine, compatible with pytorch is good

Thank you

[Nic-Ma]

Hi @Kirayue ,

Good point! The existing transforms can work for image or similar data type, but not considering the bounding box. Because seems there is no standard bounding box coordinate format yet. But, of course, it's important and valuable to support medical detection tasks in MONAI. If you have any proposal of implementation for that, welcome to submit a PR for coordinate transforms and some detection examples. Thanks.

[Kirayue]

Hi, @Nic-Ma Ref to albumentation, they use The coco format, The pascal_voc format, The albumentations format and The yolo format. In my opinion, the first two are great and general.

Do you have any suggestions for implementing coordinate transform i.e. any utils we can use or class name etc, before I dive into implementing?

[Nic-Ma]

Hi @Kirayue ,

Thanks very much for your feedback, our working group is discussing how to support detection tasks with transforms and other modules now. Let's wait for a while or you can submit a proposal for discussion if you want. Thanks.

[Kirayue]

Hi, @Nic-Ma

I don't know how detail could be called a proposal, I want to implement the bboxes transform in Dictionary-based Composables. When there is a key name called bboxes, do the bboxes transform according to the vanilla transform's parameters, like rotation angle or flip axis, etc.

If you think it's workable, I'll implement it in the few days. Or wait for a while if the working group starts to do that.

Thank you for the rapid response💯

[Nic-Ma]

Hi @Kirayue ,

I suggest developing a lightweight PR that only implements boxes support for 1 operation(like Rotate90), then let's discuss with others based on it. I really appreciate your contribution here, detection support is an important and missing piece and let's design together. Thanks.

[mibaumgartner]

@Nic-Ma @Kirayue It probably makes sense to look at already existing frameworks for medical object detection and which bounding box representation they use :)

A temporary workaround can be achieved by converting the bounding boxes into an instance-segmentation map and running that through the augmentation pipeline and convert it back at the end (this is the current approach of the medicaldetectiontoolkit. It also uses a slightly modified version of the pascal voc format (x_min, y_min, x_max, y_max, z_min, z_max) which makes support for 2D easier). I think the discussion about a uniform format will come up in the working group again :)

[Kirayue]

Hi, @Nic-Ma I made a PR for POC, the boxes format is [x_min, y_min, z_min, x_max, y_max, z_max]. I don't know which one is better, converting bounding boxes to segmentation map or process bounding box coordinates directly. But IMO, the latter needs a detailed discussion. Coordinate rotation needs a center to rotate, but in the for key in self.keys, I cannot get this information from the context.

By the way, I tried the code on the 3d_image_transforms.ipynb, I got an error spacing = Spacingd(keys=['image', 'label'], pixdim=space, interp_order=(2, 0), mode='nearest'), key named image_affine is not in the data.

here is the snippet,

[Nic-Ma]

Hi @Kirayue ,

Thanks very much for your quick PR! A good practice. I think the working group will discuss again based on your solution soon. BTW, maybe "convert to segmentation" is better, otherwise, we need to add bound box support for all the spatial transforms. Thanks.

[wyli]

possible bounding box conventions (this is in 2d only) https://github.com/albumentations-team/albumentations/blob/aa53526fdb6c789f640635a1917d4e4dd47d1069/albumentations/augmentations/bbox_utils.py#L211-L214

[wyli]

new module with tutorials: https://github.com/Project-MONAI/tutorials/tree/main/detection