Masks are not rescaled before cropping

[brandenkmurray]

In

https://github.com/broadinstitute/keras-rcnn/blob/d544263e7faef49ccd27d4240bf94c3086d57a18/keras_rcnn/preprocessing/_object_detection.py#L187-L197

the bounding box values are rescaled so the location is correct in the rescaled image. If an image is enlarged this may put the pixel locations of the box outside of the range of the original image. When the masks are read in they are cropped using the rescaled minimums and maximums above, but these are sometimes outside of the range of the mask. I believe the mask should be rescaled first, then cropped, and then resized.

https://github.com/broadinstitute/keras-rcnn/blob/d544263e7faef49ccd27d4240bf94c3086d57a18/keras_rcnn/preprocessing/_object_detection.py#L227-L241

[0x00b1]

Hi, @brandenkmurray. Thanks for the issue!

I think you’re right. We’ve been struggling with finding the best order for the generator process around yielding masks. If you’re not already working on a patch, I can commit one sometime this week.

[0x00b1]

If you’re curious about our internal thinking, here’s one (of many) proposed orderings:

• If specified, crop target image.
  • If target image was cropped, crop target mask.
• If needed, rescale target image (so one side is of length m).
  • If target image was rescaled, rescale target mask (using the scale from the previous step).
• If specified, transform (e.g. swirl) target image.
  • If target image was transformed, transform target mask.

There’s been countless debate inside our lab what’s the appropriate order for these tasks! It’s confusing and there’re way too many pros and cons for each possible ordering. 😆

cc: @jhung0 and @cells2numbers

[0x00b1]

A possible solution?

We don’t provide a crop_shape parameter in ObjectDetectionGenerator.

If target_shape is greater than image.shape, recale the image such that the smallest side of the rescaled image is equal to the minimum of target_shape then crop the rescaled image to the target_shape.

If target_shape is less than image.shape, crop the image to the target_shape.

[brandenkmurray]

I have a patch in my fork that simply rescales the mask so it matches the shape of the target image after it is read in, I will submit a pull request. I'm also unsure what the proper ordering should be but maybe it would suffice to make it a parameter and allow the user to choose the ordering, leaving the current order as the default. It would be good to test different orders on various datasets and see how they affect accuracy and speed.

[0x00b1]

Sounds good. Thanks for the patch. I merged. I'll reopen when someone feels like revisiting this. 😎